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Abstract 


A  validation  assessment  of  the  new  STORM  air-to-air  prototype  algorithm  is 
accomplished  using  structural  and  output  validation  techniques.  In  the  structural 
validation  phase,  the  algorithms,  code,  and  assumptions  are  evaluated  to  determine  if  the 
implementation  of  the  model  will  match  the  intent  of  the  designers.  The  components  of 
the  algorithm  are  compared  to  its  predecessor,  THUNDER,  to  evaluate  if  the  prototype 
improves  upon  the  weaknesses  of  THUNDER.  In  the  output  validation  phase,  the  results 
of  the  model  are  evaluated  to  determine  the  extent  to  which  the  implementation  of  the 
model  matches  expected  outcomes.  Sensitivity  analysis  is  presented  to  provide  insight 
into  the  responsiveness  of  the  algorithm  to  changes  in  aircraft  performance.  A  two-level 
half-fraction  factorial  design  with  seven  factors  is  used  to  determine  the  most  significant 
factors. 
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A  VALIDATION  ASSESSMENT  OF  THE  STORM  AIR-TO-AIR 


PROTOTYPE  ALGORITHM 


1.  Introduction 


1.1  Combat  Modeling  Tools 

Campaign  modeling  is  the  art  and  science  of  representing  the  full  theater  of 
warfare  and  the  inter-relationships  between  actors  inside,  and,  sometimes,  outside  the 
theater.  Similarly,  campaign  analysis  is  the  art  and  science  of  quantifying  the  relative 
merits  of  the  components  of  the  campaign  in  order  to  derive  insight  into  their 
performance  or  about  the  outcome  of  a  campaign  itself.  The  objective  of  campaign 
analysis  is  to  provide  information  to  senior  decision-makers  who  must  answer  the  “so 
what”  questions  involving  force  structures,  operational  concepts,  and  military 
capabilities.  The  campaign  analyst  seeks  to  measure  the  utility  and  effectiveness  of 
military  assets,  forces,  capabilities,  and  operational  concepts.  The  models  that  are  used 
for  campaign  analysis  must  be  concerned  with  breadth  before  depth  of  coverage  of  the 
campaign  environment.  A  model  that  represents  many  areas  is  more  helpful  than  a  model 
that  simulates  one  area  really  well.  At  the  campaign  level,  there  is  a  high  interest  and 
necessity  to  model  such  things  as  air-to-air  weapons  usage  and  attrition  at  a  reasonable 
level  of  fidelity. 

THUNDER  is  the  Air  Force’s  most  comprehensive  theater-level  analytical 
campaign  simulation  to  date.  It  was  designed  to  explore  issues  of  utility  and 
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effectiveness  involving  the  large-scale  application  of  air  and  space  power  in  a  joint 
warfighting  context.  THUNDER  is  typically  employed  in  applications  that  explore  issues 
of  readiness,  compare  alternative  courses  of  action,  facilitate  senior  staff  training  through 
wargaming,  and  provide  insights  into  future  military  strategies  and  evolving  operational 
concepts.  The  air  war  model  uses  a  discrete  event,  time-stepped  stochastic  simulation 
whereas  the  ground  war  model  uses  a  discrete  event,  time  stepped  deterministic 
simulation.  The  air-to-air  engagement  algorithms  used  in  THUNDER  are  based  on 
research  performed  by  George  S.  Fishman  and  Louis  R.  Moore  while  at  the  University  of 
North  Carolina  [8:8]. 

The  Synthetic  Theater  Operations  Research  Model  (STORM)  will  be  replacing 
THUNDER  as  the  primary  theater  level  model.  STORM  will  be  campaign  in  its  scope 
and  level  of  detail  but  also  “...broadens  its  purview  beyond  that  of  THUNDER,  by 
providing  more  robust  representations  of  space,  communications,  and  C4ISR  assets  and 
interactions”  [29:7]. 

STORM  will  support  in-depth  analysis  of  the  campaign-level  contributions  of  air 
and  space  power.  It  is  designed  as  a  multi-sided,  stochastic  computer  simulation  of 
military  operations  across  the  air,  space,  land,  and  maritime  domain  to  examine  issues 
involving  the  utility  and  effectiveness  of  air  and  space  power  in  a  theater-level,  joint 
warfighting  context.  STORM  originated  out  of  the  movement  of  the  Joint  world  toward  a 
new  generation  of  models.  The  Joint  community  developed  the  Joint  Simulation  System 
(JSIMS)  for  training  and  the  Joint  Warfare  System  (JWARS)  for  analysis.  To  support 
and  complement  these  tools,  each  service  is  developing  its  own  next-generation,  High 
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Level  Architecture  (HLA)-compliant  training  and  analysis  simulations.  The  National  Air 
and  Space  [Warfare]  Model  (NASM)  was  developed  by  the  Air  Force  to  address  training 
requirements.  The  National  Air  and  Space  [Warfare]  Model/Analytic,  Next  Generation 
(NASM/ AN)  program’s  objective  is  the  design  and  implementation  of  an  analytical 
simulation  of  theater-scale  model  in  a  joint  warfighting  context.  STORM  is  the 
centerpiece  of  this  effort. 

STORM  will  support  senior  decision  makers  across  the  acquisition,  policy,  and 
operations  communities.  It  will  be  able  to  provide  the  required  information  with  a  high 
degree  of  responsiveness  in  terms  of  analytical  focus  and  turn-around  time  for  results. 
Typical  uses  are  [30:2]: 

•  Analyses  of  readiness,  modernization,  sustainability,  and  force  structure 
issues. 

•  Alternative  Course  of  Action  (ACA)  studies. 

•  Assessments  of  evolving  capabilities,  alternative  strategies,  and  potential 
operational  concepts. 

•  Wargaming  to  facilitate  senior  staff  education  and  training. 

STORM  attempts  to  strike  a  proper  balance  between  functionality,  credibility,  and 
compliance.  Functionality  ensures  the  model  is  usable  by  the  analyst  through  user 
interfaces,  transparency,  available  data,  etc.  Credibility  ensures  that  the  fidelity  of  the 
simulation  accurately  represents  reality.  Compliance  ensures  that  the  model  is  developed 
with  close  regard  to  current  and  future  DoD  modeling  and  simulation  initiatives. 

STORM  fits  near  the  top  of  the  familiar  hierarchical  pyramid  of  the  Air  Force  suite  of 
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models  shown  in  Figure  1 .  Notice  that  as  one  moves  up  the  pyramid,  there  is  more 
abstraction,  larger  scope,  and  more  data  sources  [5:3]. 

As  entity  detail  decreases,  the  actions  of  the  entities  become  more  abstract.  The 
scope  refers  to  the  length  of  time  and  geography  span.  Also,  as  the  scope  increases,  the 
numbers  and  variety  of  entities  within  the  simulation  increases. 

The  air-to-air  portion  of  the  STORM  model  has  been  designed  to  improve  upon 
14  years  of  THUNDER’S  development  and  modification.  THUNDER’S  air-to-air  model 
has  been  useful  but  “...suffers  from  limiting  assumptions  that  oversimplify  the  problem” 
[29:7].  The  air-to-air  portion  is  the  focus  of  this  research. 


Figure  1.  Air  Force  Modeling  Hierarchy 
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A  THUNDER  user  must  input  the  number  and  type  of  weapons  fired  (by 
configuration,  by  mission)  for  any  air-to-air  engagement.  The  user-input  shots  tend  to 
cause  the  more  numerous  aircraft  to  overshoot  at  the  lesser  aircraft  [29:7].  Regarding 
THUNDER,  Denhard  [8:79]  identifies  four  stated  criticisms  of  the  air-to-air  submodel: 

•  Problems  with  the  single  shot  probability  of  kill  and  range  advantage  issues, 

•  Number  of  weapons  fired  per  engagement  by  an  aircraft  is  the  same  for  all  air- 
to-air  engagements  regardless  of  type  of  opponent  aircraft  faced, 

•  Multiple  weapon  salvo  is  always  modeled  as  a  SHOOT-SHOOT  firing 
doctrine  while  the  USAF  typically  employs  a  SHOOT-LOOK-SHOOT  firing 
doctrine,  and 

•  THUNDER  does  not  allow  for  aircraft  to  disengage  before  weapons  release. 

The  developers  of  STORM  have  selected  a  heuristic  and  event-oriented  approach 
to  model  air-to-air  engagements.  This  approach  was  chosen  to  improve  transparency  to 
users  and  is  more  suitable  for  medium  aggregation  needs.  Other  considered  approaches 
were  Markovian  modeling  (continuous  and  discrete).  Event  Occurrence  Networks 
(proposed  by  Denhard  for  use  with  air-to-air  engagement  modeling  [8]),  and  general 
process  modeling. 

Conceptually,  STORM’S  air-to-air  adjudicator  is  a  mixture  of  the  simple  physical 
model  and  the  interpolation  approaches.  The  simple  physical  model  provides  a 
mechanism  for  anchoring  the  concepts  of  phases,  volleys,  and  tactics.  This  statistical 
interpolation  is  designed  to  account  for  different  types  of  engagements  encountered 
during  a  campaign  without  burdening  the  user  with  vast  data  requirements.  Air-to-air 
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combat  is  broken  down  into  engagements,  which  are,  in  turn,  sectioned  into  phases.  Each 
phase  is  modeled  as  a  series  of  volleys,  with  each  volley  representing  an  opportunity  to 
make  decisions  such  as  “select  target”,  “fire”,  “disengage”,  etc.  [29:3]. 

An  engagement  occurs  between  opposing  aircraft  when  certain  conditions  exist 
such  as  detection,  cueing,  proximity,  and  intent.  The  methodology  calculates  the  weapon 
expenditures  and  aircraft  attrition  that  result  from  an  engagement.  Time,  distance, 
geometry,  and  tactics  within  the  engagement  are  not  explicitly  modeled,  and  more 
detailed  air-to-air  models,  such  a  BRAWLER,  would  be  required  to  feed  this  information 
via  input  data.  Upon  conclusion  of  the  engagement,  the  surviving  aircraft  return  to  the 
original  flight  to  determine  whether  to  continue  or  abort. 

1.2  Typical  Air-to-Air  Engagements 

Before  continuing  to  look  at  the  THUNDER  and  STORM  models  in  detail.  It 
may  be  beneficial  to  look  at  a  “typical”  air-to-air  engagement  from  start  to  finish.  This 
will  provide  a  backdrop  from  which  to  form  conclusions  whether  a  model  is  acceptable. 

The  goal  of  all  fighter  tactics  and  maneuvering  is  to  meet  your  aircraft  firing 
requirements  while  frustrating  that  of  your  enemy.  After  the  enemy  aircraft  has  been 
acquired  by  radar,  the  friendly  aircraft  will  try  to  position  his  aircraft  by  a  series  of 
maneuvers.  These  series  of  maneuvers  are  called  tactics.  In  general,  the  actual 
maneuvers  and  tactics  are  dependent  on  the  pilot’s  perception  of  the  combat  situation. 
Tactics  have  been  developed  to  take  advantage  of  the  strengths  of  one  weapon  package 
and  capitalize  on  the  weaknesses  of  the  opponent  aircraft  [8:155]. 
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USAF  aircraft  typically  engage  opponent  aircraft  in  two  or  four  ship  elements. 
Aircraft  elements  coordinate  with  each  other  concerning  tactics  and  maneuvering  and 
sharing  of  workload.  When  the  engagement  commences,  the  flights  usually  attempt  to 
maintain  only  two  ship  elements.  This  size  has  been  found  to  be  most  effective  in 
maintaining  mutual  support  while  minimizing  coordination  problems  [8:156]. 

A  typical  engagement  starts  with  40  nm  or  more  separation  between  two  opponent 
flight  groups.  Both  aircraft  are  beyond  visual  range  (BVR)  of  each  other.  An  aircraft 
group  must  intercept  the  opponent  aircraft  through  a  series  of  tactics.  These  tactics  are 
chosen  based  on  the  mission  goals,  rules  of  engagement,  the  geometry  of  the  engagement, 
and  so  on. 

For  a  tactical  intercept  there  are  six  basic  steps:  detection,  sorting,  targeting, 
intercept,  engage,  and  separate.  The  detection  phase  is  the  process  of  locating  the  enemy, 
or  bandit,  on  your  radar.  This  is  limited  by  radar  search  volume  that  encompasses 
elevation,  azimuth,  and  range.  A  pilot  and  his  wingman  normally  have  a  search  plan  for 
locating  the  target  identified  by  Ground  Control  Intercept  (GGI).  During  the  intercept 
portion  of  the  air-to-air  engagement,  the  on-board  sensors  provide  the  information 
regarding  the  opposing  flight  group.  Sorting  is  the  process  of  developing  and  updating  a 
spatial  layout  of  the  opponent.  It  requires  the  flight  to  distinguish  all  potential  targets. 
During  this  process,  questions  are  asked  about  the  state  of  the  opponent  aircraft  such  as, 
“How  many  threat  aircraft  are  out  there?”,  “What  formation  are  they  in?”,  and  “What  are 
they  doing?”.  The  friendly  aircraft  must  decide  which  aircraft  to  attack  and  then  become 
more  specific  in  their  intercept  geometry.  Targeting  involves  a  flight  taking  a  specific 
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target  of  responsibility.  Intercept  is  the  phase  where  you  actually  close  on  the  opposing 
aircraft,  trying  to  place  the  opposing  aircraft  in  weapons  envelope.  This  begins  in  the 
beyond  visual  range  (BVR)  range  when  missiles,  such  as  the  AIM- 120,  AMRAAM,  are 
within  the  weapons  envelope. 

It  should  be  understood  that  “...most  air-to-air  kills  are  against  aircraft  that  have 
no  idea  that  they  are  about  to  be  fired  upon.  The  further  away  an  aircraft  can  fire  a 
missile  at  an  opponent  and  still  have  the  missile  be  effective,  the  better”  [8:162]. 
Therefore,  this  particularly  should  be  modeled  within  the  scope  of  a  campaign  model. 

Next,  in  the  engagement  step,  the  element  enters  a  visual  fight  with  the  opponent. 
Finally,  separating  is  the  decision  process  made  in  relation  to  the  “escape  window.”  The 
escape  window  represents  the  safe  path  out  of  the  fight  or  separating  from  the  fight.  The 
following  factors  affect  your  position  in  the  escape  window  [4]: 

•  Your  range  from  the  bandit, 

•  The  energy  of  your  aircraft  relative  to  the  opposing  aircraft  (the  greater  your 
energy,  the  more  “open”  your  escape  window),  and 

•  Your  combined  angle-off  *and  aspect1 2  with  the  bandit,  with  a  head-on  pass 
giving  the  best  chance  for  an  “open”  escape  window. 


1  Angle-off  is  the  difference  between  your  aircraft  and  the  opposing  aircraft.  For  example,  if  the  angle-off 
is  0  degrees,  you  would  be  on  a  parallel  heading  with  the  opposing  aircraft.  If  the  angle-off  were  90 
degrees,  your  fuselage  would  be  perpendicular  to  the  opposing  aircraft. 

2  Aspect  is  the  number  of  degrees  measured  from  the  tail  of  a  target  to  your  aircraft.  This  angle  is 
important  because,  if  you  know  this  angle  and  range  to  target,  you  know  his  turning  room  from  the  target. 
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Perception  is  an  important  factor  in  the  air  environment.  The  task  is  to  obtain  as 
much  tactical  information  as  possible  and  analyze  the  information.  An  advantage  can  be 
gained  by  the  passing  of  information  from  one  aircraft  flight  to  another.  [8:156] 

BVR  weapons  would  be  fired  first  given  that  the  rules  of  engagement  allow  such 
action.  The  rules  of  engagement  may  require  more  information  or  visual  identification. 
Within  the  lOnm  range,  the  target  has  entered  the  within  visual  range  (WVR)  transition 
zone.  At  this  point,  each  aircraft  must  decide  whether  they  are  in  offensive,  defensive,  or 
neutral  position  relative  to  the  opponent  aircraft  and  whether  they  should  attack,  evade 
and  reengage,  or  disengage.  If  deciding  to  attack,  the  pilot  must  rely  on  combat 
maneuvering  tactics  using  infrared  missiles  and  guns. 

There  are  two  types  of  tactical  approaches  in  a  WVR  fight:  the  ‘angles’  fight  and 
the  ‘energy’  flight.  In  the  angles  approach,  the  pilot  seeks  to  turn  the  aircraft  for  a 
position  advantage  in  order  to  improve  weapon  firing  capabilities.  In  the  energy 
approach,  the  pilot  seeks  to  gain  an  energy  advantage  by  increasing  or  decreasing  the 
energy  of  the  aircraft  (e.g.  turning  radius  or  altitude  advantage)  relative  to  the  enemy 
aircraft  without  yielding  a  position  advantage.  Once  there  is  an  energy  advantage,  the 
pilot  seeks  to  establish  a  position  advantage.  Both  of  these  tactical  theories  depend  on  the 
type  of  weapon  that  is  involved  [24:99}. 

“USAF  tactics  emphasize  early  shots,  causing  disruption,  keeping  airspeed  up  and 
avoiding  getting  drawn  into  a  dogfight  type  engagement”  [8:160].  A  turning  engagement 
is  not  desired  since  pilots  would  be  fighting  independently  only  giving  mutual  support  by 
presence  [8:160],  If  it  did  come  down  to  a  turning  engagement,  basic  flight  maneuvers 
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(BFM)  describe  how  aircraft  maneuver  against  each  other.  BFM  is  usually  grouped  into 
the  following  three  categories:  Offensive,  Defensive,  and  Neutral.  With  Offensive  BFM, 
the  goal  is  to  shoot  down  the  opponent  in  the  minimum  amount  of  time.  It  requires 
placing  the  enemy  in  the  weapons  envelope  while  denying  the  opponent  an  opportunity  to 
launch.  In  Defensive  BFM,  the  goal  is  to  stay  alive  and  separate  from  the  opponent. 
Strategy  centers  around  extending  the  engagement  so  the  opponent  is  forced  into  an  error. 
With  Neutral  BFM,  this  implies  a  null  position/speed  so  that  the  pilot  can  either  separate 
or  try  to  gain  an  advantageous  position. 

1.3  Validation  Issues 

According  to  Department  of  Defense  Directive  5000.59,  validation  is  “...the 
process  of  determining  the  degree  to  which  a  model  is  an  accurate  representation  of  the 
real  world  from  the  perspective  of  the  intended  uses  of  the  model.”  In  this  definition, 
accurate  representation  of  the  real  world  is  qualified  by  the  intended  use  of  the  model. 

The  W&A  Recommended  Practices  Guide  [11 :8]  states  two  prerequisites  for  cost- 
effective  validation  -  intended  use  and  a  clear  definition  of  the  real  world.  If  one  does 
not  know  what  he  is  modeling  against,  good  and  bad  results  can  not  be  distinguished. 

The  guide  also  gives  a  “lay  man’s”  definition  of  validation  when  it  states  “...validation 
consists  of  comparing  a  prediction  (from  a  simulation)  with  an  observation  (from  the  real 
world),  and  making  a  judgment  about  whether  the  result  is  good  enough  for  application  to 
your  problem”  [11:8]. 

Air  Force  Instruction  16-1001  breaks  validation  into  two  major  components  — 
structural  validation  (also  called  “conceptual  validation”  in  other  literature)  and  output 
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validation  (also  called  “results  validation”).  Structural  validation  is  the  evaluation  of  the 
algorithms,  code,  and  assumptions  to  determine  if  the  implementation  of  the  model  will 
match  the  intent  of  the  designers.  Output  validation  is  the  evaluation  of  the  results  of  the 
model  to  determine  the  extent  to  which  the  implementation  of  the  model  matches 
expected  outcomes.  Youngblood  and  Pace’s  [33:200]  methods  parallel  that  of  the  Air 
Force  Instruction.  They  present  two  categories  of  validation  methods  -  Conceptual 
Validation  Methods  (in  general  agreement  with  the  “structural  validation”  mentioned 
previously)  and  Implementation  (Results)  Validation  Methods.  Conceptual  validation  is 
the  review  of  assumptions,  algorithms,  modeling  concepts,  data  availability,  and 
architecture  of  the  conceptual  model  to  determine  if  the  model  is  expected  to  provide  an 
acceptable  representation  of  the  subject  for  the  intended  application.  Results  validation  is 
the  review  process  that  compares  model  responses  to  known  or  expected  behavior  to 
determine  that  the  responses  are  sufficiently  accurate  for  intended  uses  [33:203]. 

Ketanni  and  Oral  [15]  present  four  major  interdependent  types  of-  formulational 
(structural),  experimental  (results),  operational,  and  data  validation.  Formulational 
validation  is  mainly  concerned  with  the  “...degree  of  relevance  of  the  assumptions  and 
theories  underlying  the  ‘formal  model’  of  the  [real  world  event]”  [15:224].  Experimental 
validation  is  concerned  with  the  “...quality  of  solutions,  the  types  of  solutions,  the  nature 
of  solution  techniques,  and  the  efficiency  of  solution  procedures  [15:224].”  Data 
validation  involves  the  “...sufficiency,  accuracy,  appropriateness,  availability, 
maintainability,  reliability,  and  cost  of  data”  [15:222].  Operational  validation  refers  to 
the  usability,  usefulness,  timeliness,  and  cost  of  implementation  of  the  model’s 
recommended  decision  and  is  beyond  the  scope  of  this  thesis. 
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This  validation  effort  will  utilize  two  methods,  structural  validation  and  results 
validation.  First,  during  the  structural  validation,  the  algorithm  for  the  air-to-air  portion 
of  STORM  will  be  reviewed  to  determine  if  it  has  adequate  fidelity  and  robustness  to 
satisfy  the  intended  use.  This  method  will  be  used  to  establish  the  scientific  basis  for  the 
algorithms  by  identifying  any  limitations  as  well  as  identifying  any  incorrect  or  restrictive 
assumptions.  Important  aspects  of  this  method  include:  identifying  the  source  upon 
which  the  algorithm  is  based,  identifying  higher-fidelity  elements  of  the  algorithm  that 
will  be  lost  because  they  are  not  processed  by  other  elements  of  the  M&S,  and  specifying 
the  accuracy  capability  of  the  individual  algorithm  [33:8].  Second,  this  validation  effort 
will  utilize  the  results  validation  method.  Comparisons  will  be  made  by  comparing  data 
points  resulting  from  THUNDER.  THUNDER  provides  a  standard  by  which  to  compare 
but  also  a  baseline  from  which  to  improve.  Also,  functional  decomposition  and  testing 
will  be  used  by  decomposing  the  STORM  air-to-air  adjudicator  into  functional 
components.  It  provides  a  means  of  performing  piecewise  testing  to  determine  if  the 
M&S  adequately  represents  the  system. 

The  stated  objectives  by  the  developers  of  STORM  are  to  “...represent  attrition  at 
least  as  well  as  THUNDER’S  current  algorithm,  improve  the  representation  of  munitions 
consumption,  develop  a  methodology  that  will  more  readily  calibrated  with  detailed 
engagement  models  (e.g.  BRAWLER),  and  still  maintains  transparency  for  analysis  in 
the  field”  [5:3].  This  thesis  effort  will  evaluate  whether  it  has  achieved  these  objectives 
and  identify  particular  strengths  or  shortcomings. 
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1.4  Modeling  and  Validation  of  an  Air-to-Air  Algorithm 

Air-to-air  adjudication  at  the  campaign  level  is  a  difficult  proposition  since  no 
accredited  approaches  exist.  Additionally,  the  problem  itself  is  complex  and  lacks  the 
data  for  the  range  of  engagements  normally  encountered  at  the  campaign-level.  STORM 
attempts  to  address  these  uncertainties  by  providing  a  flexible  aggregated  representation 
of  air-to-air  combat.  Validation  will  involve  evaluating  the  tradeoffs  between  the  degree 
of  “data  explosion”  and  transparency  to  the  user  versus  improved  fidelity  of  the  model. 
Therefore,  well-defined  conceptual  requirements  are  essential  to  ensure  that  an 
adjudicator  is  both  useful  and  usable  [29:9]. 

1.5  Statement  of  the  Problem. 

AFI 16-1001  lists  independent  third  party  validation  as  a  viable  technique  for 
model  validation.  An  independent  validation  of  the  STORM  air-to-air  algorithm  is 
necessary  to  ensure  that  it  represents  air-to-air  assets  with  an  acceptable  level  of  fidelity, 
given  the  tradeoffs  discussed  previously.  It  should  at  least  model  the  air-to-air 
environment  with  the  same  level  of  fidelity  as  its  predecessor,  THUNDER.  Furthermore, 
the  development  of  STORM  provides  an  opportunity  to  improve  the  weaknesses  of 
THUNDER.  This  validation  assessment  will  answer  the  questions,  “Have  the  lessons 
learned  from  weaknesses  in  THUNDER  been  incorporated  into  the  STORM  algorithm?”, 
“Does  the  STORM  air-to-air  adjudicator  meet  the  objectives  set  by  the  developers?”,  and, 
finally,  “Does  the  STORM  air-to-air  adjudicator  adequately  represent  reality,  keeping  in 
mind  its  intended  use?”  This  validation  effort  is  confined  to  the  use  of  the  prototype 
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developed  by  S3 1.  Therefore,  there  may  be  issues  that  are  addressed  as  deficiencies  that 
may  be  incorporated  outside  of  the  prototype  delivered. 

1.6  Thesis  Outline. 

The  approach  of  this  thesis  effort  is  to  use  validation  tools  and  techniques  to 
perform  structural/conceptual  model  validation  and  output  validation  of  the  air-to-air 
algorithm  within  STORM.  The  model  will  be  examined  to  determine  whether  or  not  it 
meets  the  objectives  stated  by  the  developers,  namely:  1)  ability  to  represent  munitions 
consumption,  2)  methodology  calibrated  with  detailed  engagement  models,  and  3) 
transparency  to  analysts  in  the  field.  This  thesis  is  organized  into  chapters  according  to 
subject  areas.  Chapter  2  presents  an  overview  of  THUNDER  with  particular  emphasis  on 
the  air-to-air  modeling  methodology.  Chapter  3  presents  an  overview  of  STORM. 
Chapter  4  is  a  comparison  of  the  THUNDER  and  STORM  theoretical  models  and  output. 
Chapter  5  presents  sensitivity  analysis  of  STORM  input  variables.  Chapter  6  presents  the 
conclusions  of  this  thesis  and  recommendations  for  improving  the  STORM  air-to-air 
adjudicator. 
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2.  THUNDER  Overview 


2.1  Introduction 

THUNDER  has  been  the  Air  Force’s  primary  theater-level  model  for  analysis 
since  1986.  It  is  a  two-sided,  stochastic  computer  simulation  of  conventional  air,  land, 
and  naval  air  warfare.  THUNDER  is  used  to  evaluate  force  structures,  conduct  Analysis 
of  Alternative  (AO A)  studies,  develop  strategies  and  tactics,  and  facilitate  senior  staff 
training  through  war-gaming.  THUNDER  will  be  replaced  by  STORM  as  the  “Air 
Force’s  campaign  analytic  tool  for  acquisition  and  course-of-action  analyses”  to  examine 
issues  of  air  and  space  power  [26:26].  This  chapter  presents  a  brief  history,  functional 
design,  and  previous  validation  efforts  of  THUNDER. 

2.2  Background  of  THUNDER 

THUNDER  was  developed  from  TAC  WARRIOR,  a  theater  level  model  used 
from  the  1970’s  through  the  early  1980’s.  TAC  WARRIOR  proved  difficult  to  use  and  to 
have  underlying  assumptions  that  were  no  longer  valid  to  meet  Air  Force  needs. 
Therefore,  the  Air  Force  Studies  and  Analyses  Agency  sponsored  the  development  of 
THUNDER  to  correct  the  shortcomings.  The  model  was  first  used  operationally  in  1986, 
as  version  2.0.  CACI,  Inc.  performed  maintenance  and  upgrades  from  1987  to  1993. 
Since  1993,  both  CACI,  Inc.  and  System  Simulation  Solutions,  Inc.  (S3 1)  have 
maintained  THUNDER.  The  most  current  version  is  THUNDER  6.6. 

THUNDER  is  written  in  SIMSCRIPT II.  5®,  a  general-purpose  programming 
language  used  for  large,  event  simulations.  The  model  consists  of  over  1,350  routines 
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combining  for  more  than  300,000  lines  of  computer  code  and  operates  on  the  UNIX 
Operating  System.  THUNDER  User  Groups  meet  yearly  to  interact  with  THUNDER 
developers  and  fellow  users.  In  addition,  basic  and  advanced  courses  are  offered  for 
those  interested.  Currently,  THUNDER  User  Groups  consist  of  35  distinct  organizations 
in  42  separate  sites  in  5  nations.  Some  notable  THUNDER  users  include: 

•  Air  Force  Studies  and  Analyses  Agency  (AFSAA) 

•  Air  Force  Wargaming  Institute 

•  Boeing 

•  Joint  Strike  Fighter  Program  Office  in  Crystal  City,  VA 

•  Republic  of  Korea  Air  Force  and 

•  Royal  Air  Force  Air  Warfare  College 

The  three  major  assumptions  identified  for  THUNDER  for  any  campaign  being 
studied  are:  1.)  The  war  is  between  two  nation-state  sized  adversaries  in  a  single  theatre 
of  operations,  2.)  A  defined  boundary  exists  between  opposing  sides  in  the  model,  and 
3.)  The  campaign  can  be  expressed  through  a  four-part  process  of  Perception,  Planning, 
Execution,  and  Adjudication. 

Although  THUNDER  is  primarily  an  air  campaign  model,  many  of  the  targets  for 
air  missions  are  generated  by  the  ground  war.  The  ground  war  cycle  of  THUNDER 
consists  of  four  sub-functions:  Command  and  unit  definition,  Initialization  of  the 
battlefield,  Rear  Area  Transportation  System,  and  Attrition  of  unit  assets.  The  primary 
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measure  of  effectiveness  (MOE)  in  the  ground  war  is  the  movement  of  the  Forward  Line 
of  Troops  (FLOT). 

Command  and  unit  definition  and  initialization  of  the  battlefield  are  scripted  into 
the  campaign  scenario  prior  to  the  start.  Ground  forces  consist  of  commands,  which  may 
have  subordinate  commands,  and  units.  The  unit  is  normally  modeled  at  the  division 
level  consisting  of  any  type  combat  unit.  Both  commands  and  units  may  possess  air 
defenses  (AD). 

The  ground  war  in  THUNDER  is  based  on  the  Center  for  Army  Analysis’s 
(CAA)  Concepts  Evaluation  Model  (CEM).  As  units  engage  in  combat  along  the  FLOT, 
combat  is  adjudicated  by  CAA’s  Attrition  Calibration  (ATCAL)  model.  Commands  and 
units  not  engaged  on  the  battlefield  may  move  along  the  rear  transportation  network 
according  to  user  input.  The  following  eight  orders  control  force  structure  and  strategies 
in  THUNDER:  SUPERIOR,  OBJECTIVE,  EXPLOIT,  FRONTAGE, 
DEPLOY.ON.CONTACT,  SECTOR,  ECHELON,  and  CAS.REQUEST. 

As  mentioned  earlier,  the  ground  war  generates  many  of  the  targets  used  in  the  air 
war.  These  targets  are  generated  by  Intelligence,  Surveillance  and  Reconnaissance  (ISR). 
The  level  of  ISR  may  be  modeled  at  low,  high,  and  very  high  resolutions.  Ground  truth 
of  enemy  positions  and  status  is  used  in  low  resolution,  while  the  perception  of  the  enemy 
deteriorates  with  time  unless  updated  with  additional  sensor  sweeps  in  high  and  very  high 
resolution  modes.  In  high  resolution  mode,  a  side’s  intelligence  on  the  enemy  is  based  on 
the  level  of  perception  over  zone-sector  areas  in  the  ISR  grid  and  only  aerial 
reconnaissance  vehicles  are  modeled.  Finally,  the  effects  of  varying  levels  of 
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reconnaissance  coverage,  timeliness,  and  sensor  quality  are  accounted  for  in 
VERY.HIGH  resolution  mode  and  space  surveillance  is  available. 

An  air/ISR  grid  and  an  associated  air  network  support  the  modeling  of  air  mission 
planning  for  air  defense  threat  avoidance.  An  air  grid  is  a  collection  of  square  area 
elements,  each  with  the  same  size.  In  THUNDER,  an  air  grid  must  have  an  even  number 
of  square  elements  in  both  the  x  and  y  coordinate  directions.  In  addition,  the  air  grid 
must  cover  the  entire  battlefield  as  specified  by  the  user;  however,  the  air  grid  may  be 
larger  and  have  different  proportions  than  the  defined  battlefield.  The  user  may  assign 
certain  squares  in  the  air  grid  to  be  no-fly  zones  for  red  and/or  blue  aircraft.  An  air 
network  is  based  on  the  air  grid  and  is  generated  by  THUNDER.  To  build  the  air 
network,  the  center  points  of  each  square  area  of  the  air  grid  are  connected  to  each  of  its 
neighbors’  center  points. 

THUNDER  simulates  three  types  of  airbases:  on-battlefield  stationary  airbases, 
off-battlefield  stationary  airbases,  and  moveable  airbases.  On-battlefield  stationary 
airbases  are  geographically  located  within  the  battlefield  grid  specifications  of 
THUNDER.  Operational  airbases  have  aircraft,  aircraft  maintenance,  logistics  resources, 
and  of  course,  runways.  Aircraft  fly  missions  against  the  enemy  from  these  bases  and, 
since  the  airbases  are  stationary,  an  airbase  may  fall  into  the  hands  of  the  enemy. 

Off-battlefield  stationary  airbases  are  positioned  off  the  defined  battlefield  and  are 
connected  to  the  air  network  via  connections  to  their  closest  square  grid  element  on  the 
air  grid.  Off-battlefield  airbases  have  the  same  resources  and  capabilities  as  the  on- 
battlefield  airbases  and  may  also  be  lost  to  the  enemy. 
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Moveable  airbases  are  carrier  battle  groups  that  model  naval  air  operations.  The 
major  impact  of  declaring  an  airbase  to  be  a  carrier  battle  group  is  that  the  airbase  can 
then  change  locations  during  the  battle. 

2.3  THUNDER  Air  War 

THUNDER  addresses  all  air  warfare  elements,  including  mission  planning,  base 
operations,  base  logistics,  flight  group  assembly,  flight  group  movement,  etc.  The  air 
war  in  THUNDER  is  driven  by  air  mission  planning. 

The  purpose  of  air  mission  planning  is  to  create  Air  Tasking  Orders  (ATOs). 
Aircraft  squadron  sorties  are  allocated  to  certain  missions  using  a  linear  program  to 
maximize  squadron  effectiveness  in  terms  of  capability,  lethality,  and  mission  priority. 
Once  the  squadrons  have  been  allocated  to  specific  missions  they  are  assigned  to  enemy 
targets  based  on  target  priorities.  Each  target  is  generated  based  upon  the  perceived  state 
of  enemy  resources  (via  ISR  discussed  earlier)  and  then  given  a  target  priority. 

THUNDER  creates  flight  groups  once  air  mission  planning  produces  the  ATOs. 
Each  flight  group  consists  of  various  flights,  one  being  a  primary  flight  with  the  same 
mission  as  the  flight  group  and  the  other  flights  acting  in  support  of  the  group  mission. 
THUNDER  allows  aircraft  flights  to  take-off  from  different  airbases  at  pre-determined 
times  and  rendezvous  at  designated  points  on  the  air  grid.  This  action  allows  for  the 
formation  of  flight  groups  from  individual  flights. 

THUNDER  also  models  air-to-air  refueling  (AAR).  At  the  beginning  of  the  air 
planning  stage,  THUNDER  determines  the  total  amount  of  air  refueling  capacity  for  each 
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air  command.  When  a  flight  leaves  an  airbase  and  heads  for  its  group  rendezvous  point, 
its  flight  path  is  not  modified  to  account  for  having  to  refuel.  The  user  may,  however, 
provide  an  appropriate  aircraft  time  delay  at  this  stage  of  the  operation  to  account  for  air 
refueling. 

Once  the  flight  groups  are  assembled,  the  groups  proceed  to  their  destinations  as 
defined  in  their  ATOs.  Aircraft  flight  paths  may  be  generated  with  the  goal  of 
minimizing  exposure  time  in  enemy  territory  or  with  the  goal  of  minimizing  vulnerability 
based  on  the  current  state  of  enemy  air  defenses.  Flights  may  then  cross  the  FLOT  to 
complete  their  missions.  At  any  time,  flights  may  incur  losses  from  enemy  ground  or  air 
threats. 

An  enemy  flight  group  may  be  detected  by  either  Ground  Controlled  Intercept 
(GCI)  radar  or  by  Airborne  Early  Warning  (AEW)  aircraft.  The  AEW  detection  range  is 
a  function  of  line  of  sight  (LOS),  AEW  Radar  Cross  Section  (RCS),  and  maximum  AEW 
radar  range. 

THUNDER  models  27  air  missions,  grouped  according  to  mission  objective  in 
accordance  with  USAF  doctrine.  There  are  six  main  mission  groups:  air-to-ground,  air- 
to-air  missions,  suppression  of  enemy  air  defense  (SEAD),  high  value  asset  (HVA),  and 
other  missions.  Table  1  provides  a  categorized  list  of  the  27  different  air  missions 
modeled.  Appendix  A  provides  a  detailed  description  of  the  “air-to-air”  missions. 
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Table  1.  THUNDER  Air  Missions 


Mission  Objective 

Mission 

Air-to-ground 

Close  Air  Support  (CAS) 

Battlefield  Air  Interdiction  (BAI) 

Air  Interdiction  (INT) 

Offensive  Counter-Air  (OCA) 

Strategic  Targets  Interdiction  (STI) 

Air-to-air 

Barrier  Combat  Air  Patrol  (BARCAP) 
Defensive  Counter- Air  (DCA) 

Over-FLOT  Defensive  Counter-Air  (ODCA) 
High  Value  Asset  Attack  (HVAA) 

Air-to-Air  Escort  (AIRESC) 

Fighter  Sweep  (FSWP) 

Suppression  of  Enemy  Air  Defenses 
(SEAD) 

Direct  SEAD  (DSEAD) 

Escort  Suppression  (ESUP) 

Escort  Jamming  (EJAM) 

Stand-off  Suppression  (SSUP) 

Close-in  Suppression  (CSUP) 

Stand-off  Jamming  (SJAM) 

Close-in  Jamming  (CJAM) 

High  Value  Asset  (HVA) 

Airborne  Early  Warning  (AEW) 

Reconnaissance  (RECCE) 

Stand-off  Reconnaissance  (SREC) 

Airborne  Refueling  (AAR) 

Offensive  Theater  Ballistic  Missiles  (OTBM) 
Defensive  Theater  Ballistic  Missiles  (DTBM) 

Other  missions 

Airlift  (LIFT) 

Hold  aircraft  in  reserve  (RESERVE) 

Move  aircraft  to  dispersal  base  if  overrun 
(DISPERSE) 

2.4  THUNDER  Air-to-Air  Functional  Design 

The  following  assumptions  of  the  air-to-air  adjudicator  have  been  listed  in  the 
draft  Accreditation  Support  Package  for  THUNDER  [2]: 

•  Given  a  successful  sensing  detection  of  enemy  aircraft  by  on-station  orbiting 
fighters  or  intercepting  fighters  at  planned  intercept  point,  an  air-to-air  event 
occurs.En-route  aircraft  do  not  initiate  air-to-air  engagements. 
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•  The  outcome  of  air-to-air  engagement  is  determined  probabilistically  as  a 
function  of  user-defined  parameters  for  aircraft/weapon  pairings  and  tactical 
rules  of  engagement. 

•  Outcomes  are  binary  for  aircraft  (dead/alive). 

•  Weapon  (missile  and/or  gun)  expenditures  and  sequence  of  firing  are  pre¬ 
defined  by  user  for  each  aircraft  configuration. 

•  When  an  air-to-air  event  results  in  a  kill,  potential  damage  to  other  aircraft  in 
the  flight  group  is  adjudicated  in  the  Support  Airbase  Operations  submodel. 

If  THUNDER  is  operating  in  high  resolution  mode,  air-to-air  combat  is  modeled 
in  considerable  detail.  Flight  groups  are  tracked  individually  and  stochastic 
determinations  are  made  as  to  whether  defender  and  threat  flight  groups  enter  into  the 
engagement.  If  an  engagement  is  joined,  then  many-on-many  engagement  models 
characterize  the  air  battle.  Attrition  rates  for  each  flight  in  the  flight  group  are 
determined  and  losses  are  incurred  based  on  tactics,  range  advantages,  and  survival  and 
kill  probabilities  for  both  the  defending  and  attacking  aircraft  types.  THUNDER 
sequentially  aggregates  the  results  of  one-on-one  combat  engagement  into  flight  group 
vs.  flight  group  combat  engagement  for  overall  attrition  values.  The  general  process  is 
depicted  in  Figure  2. 
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Figure  2.  Aggregation  of  Calculations  from  One  vs  One  to  Fit  Group 


The  following  sections  contain  a  description  of  the  air-to-air  adjudication  process 
in  THUNDER.  For  a  more  detailed  analysis  with  example,  see  [8:176-197].  The 
THUNDER  air-to-air  engagement  model,  adjudicates  combat  via  three  processes: 

•  A  process  that  calculates  a  single  shot  probability  of  kill  (SSPK)  for  each 
aircraft  against  possible  opponent  aircraft. 

•  A  process  that  aggregates  aircraft  versus  aircraft  SSPK  calculations  to  attrition 
rates  for  flight  versus  flight  and  flight  group  versus  flight  group  levels. 

•  A  process  to  assesses  losses  by  drawing  from  a  binomial  distribution. 
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2.4.1  Single  Shot  Probability  of  Kill 

Since  a  flight  in  THUNDER  is  composed  of  homogeneous  aircraft,  the  SSPK 
calculation  can  be  seen  as  a  flight  versus  flight  entity  level  process.  The  single  shot 
probability  of  kill  ( SSPK)  is  calculated  as  follows  for  each  aircraft  versus  opponent 
aircraft  combination  (each  of  these  variables  will  be  explained  in  the  following  sections): 

SSPK  =  ENG  *  LCH  *  PK  (1) 

where, 

ENG  =  the  probability  of  engaging  the  opponent  aircraft, 

LCH = the  probability  of  firing  a  weapon  at  the  opponent  aircraft,  and 
PK  =  the  probability  of  killing  an  opponent  aircraft  given  a  weapon  firing. 

2.4.2  Probability  of  Engaging  the  Opponent  Aircraft  ( ENG) 

In  THUNDER,  an  engagement  will  always  occur  if  the  intercepting  flight  group 
detects  the  target  flight  group.  The  probability  of  engaging  the  opponent  aircraft  ( ENG) 
is  based  on  the  product  of  two  factors:  the  general  engagement  probability,  ENGaew, 
(based  on  whether  the  flight  group  is  under  airborne  early  warning  (AEW)  or  ground- 
based  control)  and  a  modification  component  reflecting  the  flight  group  tactics.  ENGaew 
is  dependent  on  the  opponent’s  platform. 

The  intercepting  flight  group  can  then  employ  one  of  three  tactics: 

•  Engage  the  target  flight  group’s  escort  aircraft  only. 

•  Engage  the  target  flight  group’s  non-escort  aircraft  only. 
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•  Engage  the  non-escort  aircraft  with  percentage  a  and  escort  aircraft  with  a 
percentage  1  -  a. 

In  THUNDER,  a  is  referred  to  as  the  allocation  ratio.  This  value  is  determined  by 
calculating  a  series  of  Lanchester  difference  equations  and  an  allocation  measure  of 
effectiveness  (See  [8]  for  a  detailed  description  of  this  calculation). 

In  similar  manner,  the  target  flight  group  can  employ  one  of  three  tactics: 

•  All  escorts  split  from  the  target  flight  group  to  engage  the  intercepting  flight 
group, 

•  All  escorts  stay  with  the  target  flight  group,  or 

•  A  percentage  m  stay  with  the  target  flight  group  and  a  percentage  1  -  m  split 
from  the  target  flight  group  to  engage  the  intercepting  flight  group. 

The  m  variable  is  known  as  the  “mutual  defender  ratio”  (a  user-input  value). 

Given  these  tactics  described  above,  the  equations  for  the  probability  of  engaging 
the  opponent  ( ENG)  at  the  aircraft  (or  flight)  level  for  the  target  flights  can  be  formed. 
They  are: 

ENG  =  ENGaew*  a  (2) 

for  the  intercepting  flight  engaging  non-escorts, 

ENG  =  ENGaew  *  [(1  -  a)]  (3) 

for  the  intercepting  flights  engaging  escorts, 

ENG  =  ENGaew  *  [(1  -  a)  *  (1  -  m)  +  m]  (4) 
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for  target  flight  escorts  engaging  the  intercepting  flight,  and 


ENG  =  ENGaew  *[a*(l-m)  +  m]  (5) 

for  target  flight  non-escorts  engaging  the  intercepting  flight. 

2.4.3  Probability  of  Firing  a  Weapon  at  an  Opponent  Aircraft  (. LCH) 

The  next  variable  in  the  Single  Shot  Probability  of  Kill  calculation  is  the 
probability  of  firing  a  weapon  at  an  opponent  aircraft  (LCH).  It  is  based  on  an 
aggregation  of  the  individual  weapons  carried  by  the  aircraft  and  not  for  each  individual 
aircraft  type.  The  LCH  calculation  is  a  combination  of  the  following  variables: 

•  Relative  Range  Advantage  -  the  number  of  a  weapon  type  that  can  be  fired 
before  an  opponent  shoots  his  weapon  type, 

•  Opponent’ s  Probability  of  Engagement, 

•  Number  of  friendly  and  opponent’s  aircraft  in  each  flight, 

•  Opponent’s  Probability  of  Kill,  and 

•  Opponent’s  “Force  Multiplier”. 

In  order  to  fire,  one  of  the  following  three  scenarios  must  exist  for  the  firing 
aircraft: 

•  The  firing  aircraft  has  an  overall  relative  range  advantage  and  can  fire  without 
threat  of  being  killed  by  the  opponent  (i.e.  the  firing  aircraft  can  fire  at  the 
opponent  before  the  opponent  fires  a  weapon  at  it), 
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•  The  firing  aircraft  does  not  have  an  overall  relative  range  advantage,  but  the 
opponent  aircraft  cannot  engage  the  firing  aircraft,  or 

•  The  firing  aircraft  does  not  have  an  overall  range  advantage,  but  the  firing 
aircraft  survives  a  specified  number  of  firings  of  an  aggregate  weapon  by  the 
opponent. 

Each  of  these  has  a  probability  of  occurrence  that  is  added  together  to  determine 
LCH.  Let  each  of  the  probabilities  for  the  above  scenarios  be  designated  by  LCHj ,  i  =1, 
2,  3  respectively.  Thus,  LCH  =  LCH )  +  LCH2  +  LCH 3 

For  the  first  scenario,  LCH}  can  best  be  understood  by  an  example.  Consider  an 
F-15C  versus  a  MIG-29.  The  F-15C  is  carrying  AIM- 120s  and  AIM-9s.  The  MIG-29  is 
carrying  AA-lOs  and  AA-8s.  There  are  four  possible  combinations  of  first  weapons  fired 
between  these  two  aircraft: 

(1)  AIM- 120  versus  AA-10, 

(2)  AIM- 120  versus  AA-8, 

(3)  AEM-9  versus  AA-10,  or 

(4)  AIM-9  versus  AA-8. 

In  combinations  1  and  2,  the  F-15C  has  a  range  advantage  over  the  MIG-29.  In 
combination  3,  the  MIG-29  has  a  range  advantage  over  the  F-15C.  In  combination  4, 
neither  side  has  a  range  advantage.  Assuming  that  the  weapon  combinations  are  random, 
the  F-15C  will  have  a  single  range  advantage  over  the  MIG-29  50%  of  the  time.  The 
MIG-29  will  have  a  single  range  advantage  over  the  F-15C  25%  of  the  time.  Neither 
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aircraft  will  have  a  relative  range  advantage  25%  of  the  time.  Therefore,  LCHi  for  the  F- 
15C  is  0.50.  LCHi  for  the  MIG-29  is  0.25. 


The  probability  for  scenario  two,  LCH2,  depends  on  the  occurrence  of  two 
independent  events.  First  is  the  firing  aircraft  does  not  have  a  relative  range  advantage  on 
the  opponent  aircraft.  The  probability  of  this  event  is  1  -  LCHi.  Second,  is  that  the 
opponent  aircraft  does  not  engage  the  firing  aircraft.  The  probability  of  this  event  is  1  - 
ENG0pPonent •  LCH2  is  then  defined  as: 

LCH2  =  [1  -  LCHi]  *  [1  -  ENGopponent]  (6) 

The  probability  for  scenario  three,  LCH3,  depends  on  the  occurrence  of  the 
following  three  events.  First,  the  firing  aircraft  does  not  have  a  relative  range  advantage 
on  the  opponent  aircraft.  Second,  the  opponent  aircraft  engages  the  firing  aircraft  with 
the  probability  of  this  event  denoted  by  ENGopp0nent •  Lastly,  the  firing  aircraft  survives  n 
weapon  firing  from  the  opponent  aircraft  before  his  first  weapon  firing.  The  probability 
of  this  event  is  equal  to: 

(1  -PKf  (7) 

where  PK  is  the  probability  that  the  opponent  aircraft  kills  the  firing  aircraft  first.  The 
PK  calculation  will  be  described  in  the  next  section.  The  number  of  weapon  firings,  n,  of 
the  opponent  aircraft  is  defined  as: 


,  n 

n  —  FM  ■  — 


(8) 


where, 


nA  =  The  number  of  opponent  aircraft, 
nB=  The  number  of  firing  aircraft, 

FM  =  X  Z  min([^&i N '  FLT  ]  •  FLT  ■  FL?C ) which  is  referred 

i  j 

to  as  the  force  multiplier, 

RRA  =  Relative  Range  Advantage  (the  number  of  shots  that  the 
opponent  can  fire  before  your  aircraft  can  fire), 

FLopp  =  the  fraction  of  launches  for  the  opponent  aircraft, 

Flfac  =  the  fraction  of  launches  for  the  firing  aircraft,  and 

N  =  the  number  of  weapon  firings  the  opponent  may  take. 

The  concept  of  fractional  launches  can  be  explained  by  the  following  example. 
The  aggregate  weapon  for  a  F-15C  with  an  equal  combination  of  AIM-120s  and  AIM-9s 
will  have  50%  of  the  characteristics  of  the  AIM-120  and  50%  of  the  characteristics  of  the 
AIM-9.  These  characteristics  are  called  “fractional  launches”  in  THUNDER.  Assume, 
for  another  example,  that  a  MIG-23  is  firing  2  AA-7  missiles  and  2  AA-2  missiles.  The 
fractional  launches  of  the  AA-7  is  2/4  or  1/2,  and  the  fractional  launches  of  the  AA-2  is 
also  2/4  or  1/2. 

2.4.4  Probability  of  Killing  an  Opponent  Aircraft  {PK) 

The  user  inputs  the  maximum  number  of  launches  per  engagement.  The 
engagement  model  assumes  that  an  aircraft  will  always  fire  its  maximum  number  of 
launches  per  engagement  unless  there  is  an  insufficient  number  of  weapons  remaining. 
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The  user  determines  the  munitions  fired  in  the  1st,  2nd,3rd,...,  nth  engagement.  As 
mentioned  previously,  all  weapons  fired  in  a  single  engagement  by  a  single  aircraft  are 
aggregated  into  a  composite  weapon.  The  PK  values  in  THUNDER  are  weapon  versus 
platform  dependent.  To  illustrate  the  PK  calculation,  assume  you  have  MIG-29s  (i  =  A) 
with  an  AA-10  (j  =  1)  and  AA-8  (j  =  2)  and  F-15Cs  (i  =  B)  with  an  AIM-120  (j  =  1)  and 
AIM-9  (j  =  2).  The  composite  probability  of  kill  calculation  is  as  follows: 


PKa  =  [FLA]  *  PKAi ]  +  [FLA2  *  PKA2 ] 

(9) 

PKb  =  [. FLb 1  *  PKb j]  +  [FLb2  *  PKb2] 

(10) 

where, 

FLlj  -  the  fractional  launches  where  i  is  the  aircraft  type  and  j  is  the 
weapon  type, 

PK  j  =  the  probability  of  kill  for  each  aircraft/weapon  combination. 

2.4.5  Attrition  Calculations 

Next,  the  aircraft  versus  aircraft  SSPK  calculations  are  aggregated  to  determine 
attrition  rates  for  flight  versus  flight  and  flight  group  versus  flight  group  levels.  Two  sets 
of  calculations  are  performed  for  the  flight  versus  flight  combinations.  First,  it  calculates 
the  attrition  rate  of  a  single  opponent  aircraft  when  engaged  by  the  entire  firing  flight 
firing  one  weapon  each.  Second,  it  calculates  the  attrition  of  the  single  opponent  aircraft 
when  engaged  by  the  entire  firing  flight  group  firing  all  weapons. 

2.4.5.1  Firing  Flight  Firing  One  Weapon  Each 

Allowing  for  the  possibility  of  multiple  kills,  the  probability  of  j  out  of  i  firing 
aircraft  killing  an  opponent  aircraft  is: 
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(11) 


UJ 


(1  -  SSPK)'-J  ■  SSPK 


THUNDER  assumes  a  weighted  combination  of  the  two  firing  flight  engagement 
strategies  based  on  the  degree  of  command  and  control  of  the  flight.  First,  if  the  firing 
flight  has  perfect  command  and  control  (FCC=  1),  aircraft  in  the  firing  flight  would 
divide  themselves  evenly  over  the  aircraft  in  the  opposing  flight.  Second,  if  there  was  no 
command  and  control  (FCC  =  0),  aircraft  in  the  firing  flight  would  divide  themselves 
randomly  over  the  aircraft  in  the  opponent  flight.  In  the  perfect  command  and  control 
case  ( FCC  =1),  the  probabilities  that  M  and  M+l  aircraft  from  the  firing  flight  engage  an 
aircraft  from  the  opponent  flight,  Fm  and  Fm+  i  respectively,  are: 


FM+l  =  REMAINDER 


K°PPj 


(12) 


M 


1  -F 


M+l 


(13) 


where, 

fac  =  number  of  firing  aircraft,  and 
opp  =  number  of  targeted  aircraft. 

If  there  is  no  command  and  control  (FCC  =  0),  the  attacking  aircraft  randomly 
attack  the  defending  aircraft.  There  would  then  be  the  possibility  that  some  aircraft 
would  not  be  attacked,  while  others  would  be  attacked  by  several  different  aircraft.  The 
probability  that  a  defending  aircraft  will  be  attacked  exactly  i  times  is: 
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\fac-i 


(14) 


G.  = 


°PP) 


1 


1  — 
k  °PPJ 


fad 


i\{fac-i ) 


In  reality,  the  command  and  control  is  between  these  two  extremes  of  perfect 
command  and  control  to  complete  lack  of  command  and  control.  The  probability  that 
defending  aircraft  in  a  flight  will  be  attacked  exactly  0,  1,2,  ...n  times  where  n  is  the 
number  of  aircraft  in  the  attacking  flight  is: 


Pi=Fi  ■  FCC  +  (1  -  FCC)  ■  Gt 


(15) 


where, 


FCC  =  the  flight  command  and  control  (average  of  the  aircraft  command  and 
control  values  in  the  flight,  a  number  between  0  and  1),  and 


Ft  =  from  (12)  and  (13)  discussed  previously,  the  probability  that  i  aircraft  from 
the  firing  flight  engage  an  aircraft  from  the  opponent  flight  under  the  strategy 
of  “perfect”  command  and  control. 

Therefore,  for  each  flight  versus  flight  combination,  the  attrition  rate  of  a  single 
aircraft  in  the  opponent  flight  entity  when  engaged  by  the  entire  firing  flight,  firing  one 
weapon  each,  is: 


ATTR  =  1*1 

i  j 
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(16) 


2.4.5.2  Firing  Flight  Group  Fires  All  Weapons 

This  calculation  is  similar  to  the  entire  flight  firing  one  weapon  each.  Two 
probabilities  are  calculated: 


32 


•  The  probability  that  the  opponent  flight  will  be  engaged  i  times  by  the  firing 
flight  (i  =  (1  ...m),  where  m  is  the  maximum  number  of  weapon  firings  for  the 
firing  flight. 

•  The  probability  that  in  k  out  of  i  engagements,  the  firing  flight  kills  and 
opponent  aircraft. 


Again,  like  the  calculation  of  the  firing  of  one  weapon  each,  THUNDER  assumes 
a  weighted  combination  of  two  firing  flight  engagement  strategies  based  on  the  degree  of 
command  and  control  of  the  flight.  It  is  also  based  on  the  maximum  number  of  launches 
any  aircraft  in  the  flight  group  can  make  (max).  First,  if  the  firing  flight  has  perfect 
command  and  control  (FGCC  =1),  aircraft  in  the  firing  flight  would  divide  themselves 
evenly  over  the  aircraft  in  the  opposing  flight.  Second,  if  there  was  no  command  and 
control  (FGCC  =  0),  aircraft  in  the  firing  flight  would  divide  themselves  randomly  over 
the  aircraft  in  the  opponent  flight.  In  the  perfect  command  and  control  case  (FGCC  =1). 
A  flight  group’s  FGCC  value  is  assumed  to  be  the  average  of  all  FCC  values  of  the 
flights  in  the  flight  group. 

The  two  probabilities  of  engagement,  Fm  and  Fm+ 1,  that  must  be  computed  for  the 
flight  group  are: 


F™ ,  =  REMAINDER^  ^ 

\opp) 

Ffg  =  1  -  F 

r  M  1  1  M  +1 


(17) 


(18) 


where, 
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fac  =  number  of  firing  aircraft,  and 


opp  =  number  of  targeted  aircraft. 

If  there  is  no  command  and  control  ( FGCC  =  0),  the  attacking  aircraft  randomly 
attack  the  defending  aircraft.  The  probability  that  a  defending  aircraft  will  be  attacked 
exactly  i  times  is: 
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(19) 


The  probability  that  defending  aircraft  in  a  flight  group  will  be  attacked  exactly  0, 
1,  2,  ...n  times  where  n  is  the  number  of  aircraft  in  the  attacking  flight  group  is: 

pFG  =  ffg  .  FGCC  +  (1_  FGCC)  •  Gfa  (2°) 

where, 

FGCC  =  the  flight  group  command  and  control  (average  of  the  aircraft  command 
and  control  values  in  the  flight  group,  a  number  between  0  and  1),  and 


Ffa  =  the  probability  that  i  aircraft  from  the  firing  flight  group  engage  an 

aircraft  from  the  opponent  flight  under  the  strategy  of  “perfect”  command 
and  control. 

Therefore,  for  each  firing  flight  group  versus  opponent  flight  combination,  the 
attrition  rate  of  a  single  aircraft  in  the  opponent  flight  when  engaged  by  the  entire  flight 
group  firing  all  available  weapons  is: 
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(21) 


ATTRfg 
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In  order  to  ensure  that  the  flight  group  attrition  rates  are  less  than  or  equal  to  one, 
the  attrition  rates  are  “normalized.”  This  is  done  by  multiplying  the  attrition  rate  by  the 
ratio  of  the  total  number  of  firings  by  the  firing  flight  over  the  number  of  total  firings  by 
the  flight  group. 


2.4.6  Engagement  Losses  Calculations 

For  each  opponent  flight,  the  number  of  aircraft  lost  is  calculated  using  a  draw 
from  a  binomial  distribution.  Because  the  actual  aircraft  losses  cannot  exceed  the 
weapon  firings,  the  actual  amount  of  aircraft  lost  is  the  minimum  of  the  binomial  draw 
and  the  number  of  weapon  firing  designated  for  the  firing  flight.  Similarly,  because  the 
number  of  aircraft  losses  cannot  exceed  the  number  of  aircraft  in  the  opponent  flight,  the 
total  amount  of  opponent  aircraft  lost  is  the  minimum  of  the  sum  of  actual  aircraft  losses 
for  each  firing  flight  and  the  number  of  aircraft  in  the  opponent  flight.  This  assessment  is 
performed  for  each  flight  in  the  opponent  flight  group. 


2.5  THUNDER  Current  Validation  Efforts 

According  to  the  Joint  Accreditation  Support  Activity  [2:4-4], 


As  a  campaign-level  simulation  of  military  operations,  THUNDER  is  in  a 
category  of  M&S  for  which  output  validation  options  are  limited  at  best.  The 
general  consensus  within  the  W&A  community  is  that  the  complexity  of  the 
known/addressed  factors  (the  numbers  and  types  of  objects,  processes  and 
actions)  and  the  extent  of  the  unknown/unaddressed  factors  (behaviors,  effects 
and  interactions),  when  combined  with  lack  of  relevant  field  data,  make  output 
validation  extremely  difficult  if  not  impossible  in  the  theoretical  sense. 
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THUNDER  has  been  used  in  numerous  studies  by  many  different  organizations. 
These  studies  are  acknowledged  by  JASA  to  represent  face  validation  that  is 
“...significant,  but  generally  not  well-documented”  [2:4-4].  In  an  effort  to  organize  the 
studies  that  use  THUNDER,  a  database  has  been  developed  to  track  the  users  and  studies 
that  benefited  from  THUNDER  output.  The  intention  was  to  support  THUNDER’S 
conceptual  model.  It  is  organized  so  that  one  can  chose  a  submodel  within  THUNDER  to 
see  all  of  the  studies  that  focused  on  that  particular  part  of  THUNDER. 

Validity  of  input  data  is  an  important  aspect  of  output  validity.  Documentation  of 
data  sources  (i.e.,  data  pedigree)  is  being  developed  by  some  THUNDER  users,  e.g., 
ASC/XR  for  the  JSF  program.  This  will  facilitate  subsequent  data  validation  efforts. 
Another  extensive  validation  effort  was  accomplished  on  the  Intelligence,  Surveillance, 
and  Reconnaissance  Module  by  Nelson,  1998  [21]. 

2.6  Summary 

In  addition  to  the  features  reviewed  here,  several  improvements  and  modifications 
to  THUNDER  6.6  have  been  recommended/proposed.  For  instance,  in  the  air-to-air 
submodel,  it  has  been  requested  that  there  be  the  ability  to  shoot  through  clouds  or  a 
certain  altitude  limitation,  referred  to  as  a  “harddeck”.  This  harddeck  would  be  specified 
as  an  aircraft  planning  factor  (by  type  aircraft,  by  time).  It  will  put  limitations  on  the 
aircraft's  profile  when  it  is  required  to  deliver  below  ceiling.  Also,  this  would  allow  for 
the  real-world  situation  that  air  defense  systems  cannot  acquire  targets  above  weather 
ceiling. 
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THUNDER  is  a  large  and  complex  model  that  requires  continual  updating  to 
improve  its  ability  to  model  the  warfare  environment.  Despite  some  limitations,  the 
usefulness  of  the  model  cannot  be  overstated.  There  is  much  that  can  be  learned  from  the 
analysis  of  various  weapon  types,  capabilities,  strategies,  and,  especially,  their 
interactions.  A  strength  of  THUNDER  is  that  the  Air  War  is  modeled  at  a  slightly  higher 
resolution  than  is  available  from  other  resources,  while  the  automatic  ATO  generator 
saves  the  user  time  to  input  the  orders. 

Despite  its  usefulness,  many  believe  the  time  has  come  for  the  next-generation 
campaign  model  to  be  developed.  STORM  will  be  used  to  capitalize  on  THUNDER’S 
strengths  and  improve  its  limitations.  In  particular,  THUNDER’S  air-to-air  adjudicator 
will  be  redesigned  to  model  attrition  as  accurately  as  THUNDER  but  improve  such  things 
as  munitions  consumption. 
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3.  Review  of  the  STORM  Air-to-Air  Algorithm 


3.1  Introduction 

The  National  Air  and  Space  [Warfare]  Model  is  the  parent  model  of  the 
NASM/AN  Program.  The  main  focus  of  NASM  is  on  training,  but  the  Operational 
Requirements  Document  requires  a  robust  analytical  capability  to  measure  the 
contributions  of  air  and  space  power  at  the  campaign  level.  Therefore,  STORM  is  the 
centerpiece  effort  to  design  the  next  generation  analytical  simulation  of  theater-scale 
military  operations.  System  Simulations  Solutions  (S3 1)  is  the  primary  model  developer 
and  system  integrator. 

The  developers  of  STORM  have  selected  a  heuristic  and  event-oriented  approach 
to  modeling  air-to-air  engagements.  This  approach  was  chosen  in  order  to  make  the 
process  more  transparent  to  users  and  suitable  for  medium  aggregation  needs.  STORM 
attempts  to  address  the  uncertainties  of  various  air-to-air  situations  by  providing  a 
flexible  aggregated  representation  of  air-to-air  combat. 

3.2  STORM  Overview 

Regarding  model  development,  the  STORM  initial  approach  is  to  cover  a  wide 
spectrum  of  missions  and  representation  with  a  limited  level  of  detail.  As  development 
continues,  specific  mission  areas  will  be  added  as  appropriate.  “The  candidates  for 
increased  detail  include  those  missions  most  critical  to  the  proper  representation  of  air 
and  space  power  and  missions  most  understood  by  operators  and  subject  matter  experts 
[26:12].” 
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STORM’s  core  model  will  be  an  event-driven,  stochastic  process  in  which  entities 
from  multiple  sides  will  interact  in  air,  land,  sea,  and  space  environments.  It  is  being 
written  in  the  C++  programming  language  and  will  be  compliant  with  DoD  high  level 
architecture  (HLA)  requirements. 

There  are  five  top-level  object  classes  in  STORM:  Environment,  C2  Managers, 
Interaction  Managers,  Assets,  and  Intelligence  Managers.  The  Environment  is  the 
foundation  of  STORM  and  serves  as  the  game  board  for  the  entities.  It  provides  weather 
and  terrain  effects  for  the  rest  of  the  model’s  objects.  Weather  is  divided  into  forecasted 
and  actual  weather.  Forecasted  weather  is  used  in  the  planning  process  while  actual 
weather  affects  detection,  attrition  and  weapon  delivery  events.  Terrain  is  the  medium 
upon  which  surface  entities  move  and  affects  how  the  C2  Managers  plan  for  entity 
movements.  Terrain  also  affects  weapon  lethality,  direct  vs.  indirect  fire  potential,  terrain 
masking  of  sensors,  etc.  The  C2  Managers  provide  computer  control  of  the  planning 
processes  for  the  simulation  entities.  An  analyst  must  “teach”  the  computer  C2  Manager 
to  make  decisions  that  adapt  to  the  warfare  simulation.  The  C2  Manager’s  plans,  in  turn, 
control  the  core  behavior  of  each  entity  in  STORM.  Interaction  Managers  are  the 
“referees”  for  STORM  and  adjudicate  any  interactions  between  entities.  In  order  to 
facilitate  simulation  development,  modification,  and  enhance  model  transparency, 
STORM  has  separated  interactions  into  their  own  objects  (e.g.  Space,  Air,  Ground,  etc.). 
Assets  are  explicit  representations  of  the  entities  within  the  simulation.  These  assets  are 
divided  into  object  sub-classes  which  are  given  individual  characteristics  through  input 
data.  These  assets  are  aggregated  and  disaggregated  dynamically  to  apply  the  appropriate 
amount  of  detail  while  meeting  runtime  constraints.  The  capstone  of  STORM,  the 
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Intelligence  Managers,  control  the  perception  of  the  entities  within  the  simulation  (i.e.  the 
ground  truth).  The  Intelligence  Manager  logs  and  updates  information  as  it  is  perceived. 

It  also  reconciles  the  degree  of  overlap  and  redundancy  as  multiple  observations  are  made 
of  numerous  entities.  It  combines  all  observations  of  all  entities  into  a  single  picture  of 
the  battlespace.  “This  perception  employs  the  ‘best’  observation  of  each  ‘known’  entity 
to  develop  an  information  base  to  support  the  C2  Managers’  planning  functions  and  the 
reactions  of  individual  Assets  during  the  simulation”  [26]. 

Following  USAF  doctrine,  STORM  missions  are  defined  by  their  objectives 
rather  than  the  type  of  aircraft  accomplishing  the  mission.  Appendix  A  contains  a 
detailed  description  of  the  mission  types.  Table  2  lists  STORM’s  planned  air-to-air 
mission  capabilities  and  their  ties  to  Air  Force  doctrine  [30:59]. 

3.3  STORM  Air-to-Air  Methodology 

Certain  aspects  of  air-to-air  combat  are  modeled  outside  of  the  air-to-air 
adjudicator.  For  instance,  the  initial  STORM  air-to-air  prototype  assumes  that  an 
engagement  can  occur.  The  steps  that  lead  up  to  an  engagement  are  assumed  to  have 
already  occurred.  The  actual  ATO  generation  and  routing  of  aircraft  according  to 
mission  type  is  outside  the  scope  of  this  thesis  project. 

STORM  models  time  and  space  implicitly  by  describing  time  and  space  in  terms 
of  a  discrete  hierarchy.  Air-to-air  combat  is  broken  down  into  engagements,  and  these 
engagements  are  broken  down  into  phases.  The  phases  are  composed  of  a  series  of 
volleys  so  that  “...weapons  expenditure  and  attrition  can  be  calculated  with  higher  fidelity 
and  air-to-air  analysis  can  be  more  transparent  [28:9].”  Therefore,  the  goal  of  the 
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STORM  methodology  is  to  remain  “campaign”  in  its  scope  while  drawing  on 
“engagement”  level  principles  to  improve  fidelity. 

The  air  motion  and  detection  manager  object  triggers  the  engagement  outside  of 
the  air-to-air  adjudication  manager  object.  The  air-to-air  algorithm  calculates  the  weapon 
expenditure  and  attrition  that  result  from  the  engagement.  The  air-to-air  engagement 
process  is  shown  graphically  in  Figure  3  and  will  be  discussed  in  detail  in  the  following 
sections. 


Table  2.  STORM’s  Air-to-Air  mission  capabilities  as  tied  to  Air  Force  Doctrine 


STORM  Mission 
Description 

Air  Force  Doctrinal  Mapping 

Core  Competency 

Function 

Element 

Fighter  Sweep 

Air  &  Space 
Superiority 

Counter  Air 

Offensive  Counter  Air 

Ground  Alert  OCA 

Air  &  Space 
Superiority 

Counter  Air 

Offensive  Counter  Air 

Fighter  Escort 

Air  &  Space 
Superiority 

Counter  Air 

Offensive  Counter  Air 

Offensive  BARCAP 

Air  &  Space 
Superiority 

Counter  Air 

Offensive  Counter  Air 

Defensive  BARCAP 

Air  &  Space 
Superiority 

Counter  Air 

Defensive  Counter  Air 

Ground  Alert  DCA 

Air  &  Space 
Superiority 

Counter  Air 

Defensive  Counter  Air 

HVA  protection 

Air  &  Space 
Superiority 

Counter  Air 

Defensive  Counter  Air 
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3.3.1  Engagement  Initialization 

The  engagement  begins  with  the  initial  tactical  maneuvering  of  two  opposing 
flight  groups.  The  flight’s  aircraft  are  broken  down  into  elements,  with  a  user-defined 
minimum  number  of  aircraft  per  element.  Weapons  are  allocated  in  a  uniform  fashion  in 
cases  where  full  loads  do  not  exist.  The  engagement  generates  three  possible  outputs  - 
weapon  expenditure,  attrition,  or  an  escape.  At  the  end  of  the  engagement,  the  surviving 
aircraft  and  unexpended  weapons  are  returned  to  the  original  flights  where  mission  logic 
(outside  of  the  prototype)  determines  the  package’s  behavior  [28:10]. 


Initialize 

Initialize 

Initialize 

Select  Flight 

- 

Engagement 

- p 

Phase 

- p 

Volley 

W 

Group  Targets 

W 

Select  Flight 
Group  Weapons 


Allow  Escapes 

— ► 

Select  Next 

x  r_.il  __  7 

— ► 

Exchange 

Volley 

— ► 

End  Phase 

V  olley 

Figure  3.  STORM  Air-to-Air  Methodology 
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3.3.2  Phase  Initialization 


As  mentioned  previously,  phases  are  used  to  capture  time  and  space  without 
explicitly  modeling  these  concepts.  These  phases  are  controlled  based  on  specific  input. 
Other  user-input  data  will  be  influenced  by  the  definitions  of  these  phases.  Notice  in 
Appendix  B  -  D  that  the  phases  are  used  as  the  primary  index  for  inputting  data 
requirements.  For  example,  when  the  probability  of  weapons  effectiveness  is  needed  for 
a  particular  weapon  firing,  the  program  determines  the  phase  that  is  in  progress,  finds  the 
appropriate  phase  under  the  probability  of  weapons  effectiveness  category  (Appendix  C), 
and  locates  the  probability  for  that  particular  weapon  type. 

In  the  STORM  air-to-air  prototype,  the  designers  have  incorporated  three  phases  - 
very  long  range  (VLR),  beyond  visual  range  (BVR),  and  within  visual  range  (WVR). 
These  phases  are  sequential  with  a  probability  of  escape,  Pe,  at  each  stage.  Unlike 
BRAWLER,  “looping  back”  into  a  previous  stage  is  not  allowed.  A  comparison  of 
STORM’s  and  BRAWLER’S  phases  are  depicted  in  Figure  4. 


STORM:  Sequential  -  Only  path  is  from  left  to  right 


BRAWLER:  Not  Necessarily  Sequential  -  Reentry  into  earlier  phases 


Figure  4.  STORM  Versus  BRAWLER  Phase  Paths  f24:15J 
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The  phase  concept  is  an  innovative  way  of  addressing  the  air-to-air  combat 
environment  for  a  campaign  level  model.  Transparency  is  improved  because  tracking  of 
combat  events  is  improved.  One  word  of  caution  though  -  because  of  eventual  wide 
dissemination  of  the  model,  the  definition  of  phases  should  be  determined  and  agreed 
upon.  Since  the  phases  are  the  framework  upon  which  the  whole  model  is  built,  there 
should  be  some  standardization  of  the  distances  that  define  a  phase.  One  possible 
approach  is  to  use  BRAWLER  to  construct  scatter  plots  of  time  and  distance  of  firings. 
This  reveals  clusters  of  missile  activity  that  could  be  used  to  define  a  phase  [6],  These 
definitions  should  then  be  agreed  upon  by  the  user  community. 

During  some  of  the  calibration  efforts,  the  developers  have  initially  defined  the 
phases  in  the  following  way.  For  the  BVR  phase,  they  assumed  that  all  shots  fired  from 
one  aircraft  in  fewer  than  20  seconds  were  part  of  the  same  volley.  In  another  phase,  the 
time  period  was  set  to  1 0  seconds.  The  time  distinction  was  made  to  account  for  the  fact 
that  at  longer  ranges,  missile  flight  times  are  longer,  and  therefore  so  are  periods  between 
pilot  assessments,  and  hence  volleys  [29:17]. 

There  are  issues  concerning  time  and  space  that  arise  in  a  realistic  air-to-air 
combat  engagement  that  should  also  be  taken  into  account,  either  explicitly  or  implicitly. 
When  time  and  distance  are  modeled  in  this  event  hierarchy,  limitations  that  arise  from 
fuel  limitations  must  be  considered.  Also,  the  position  where  the  engagement  occurs 
affects  how  the  attackers  and  defenders  respond  to  each  other.  Aircraft  positions  relative 
to  the  FLOT  affect  flight  tactics  and  decisions.  The  final  implementation  of  the 
algorithm  into  STORM  should  account  for  these  issues. 
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3.3.3  Computation  of  Answered/Unanswered  Volleys 

STORM  disaggregates  the  engagements  into  a  series  of  volleys  that  make  up  a 
particular  engagement.  This  more  explicit  modeling  concept  allows  for  a  more  accurate 
representation  of  munitions  consumption  and  firing  doctrine  since  each  volley  is 
separated  by  an  assessment  of  the  opponent’s  forces  resulting  from  your  attack. 

Each  of  the  three  phases  are  subdivided  into  sets  of  discrete  volleys.  A  volley  is 
defined  as  an  opportunity  for  aircraft  to  fire  one  or  more  salvos  of  one  or  more  weapons 
against  one  or  more  targets. 

There  are  two  types  of  volleys,  answered  and  unanswered.  An  answered  volley  is 
where  weapons  are  exchanged  and  each  side  suffers  attrition.  An  unanswered  volley  is 
where  only  one  side  fires  and  only  the  other  side  suffers  attrition.  The  unanswered  volley 
methodology  is  used  to  capture  aspects  of  surprise  of  aircraft  and  range-advantages  of 
certain  weapons.  This  is  a  significant  feature  given  the  statistic  that,  historically,  80%  of 
all  aircraft  shot  in  a  real-life  air-to-air  engagement  never  detected  the  opponent  that  killed 
them  [4:24]. 

Given  that  an  opportunity  for  an  unanswered  volley  exists,  the  algorithm  first 
computes  which  side  shoots  the  unanswered  volley.  This  is  determined  by  a  ratio  of  user- 
defined  probabilities  that  each  side  shoots  the  unanswered  volley.  That  ratio  is, 

_ Pr(side  1  shoots  unanswered  volley) _  (22) 

Pr(side  1  shoots  unanswered  volley)  +  Pr(side  2  shoots  unanswered  volley) 

If  this  ratio  is  less  than  an  pseudo-random  generated  number  (uniformly  distributed),  then 
side  1  shoots  the  unanswered  volley. 
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The  probability  of  an  unanswered  volley  is  entered  by  the  user  in  the  A2A.DAT 
file  shown  in  Appendix  C.  It  is  a  function  of  phase,  weapon,  side’s  tactic  (for  instance  - 
averse,  medium,  or  tolerant),  and  opponent’s  weapon.  For  example,  consider  a  fighter 
with  a  probability  of  0.3  in  the  VLR  phase  with  an  AIM-54  with  “tolerant”  tactics  versus 
the  opponent’s  AA-9  munition.  Further,  suppose  the  opponent’s  probability  of  shooting 
his  AA-9  in  VLR  stage  with  “averse”  tactics  versus  the  AIM-54  is  0.1 .  The  calculation 
would  result  in  a  ratio  of  0.75.  Thus,  side  1  would  have  a  75%  chance  of  shooting  an 
unanswered  volley. 

The  probability  of  an  unanswered  volley  has  the  versatility  to  be  tied  to  a 
particular  platform.  This  is  fortunate  because  it  should  take  into  account  the  on-board 
systems  when  determining  this  probability.  An  AIM- 120,  for  instance,  should  have  a 
different  probability  firing  from  an  F-15  than  an  F-16.  The  reader  should  begin  to  see  the 
additional  data  burden  that  comes  with  the  added  versatility.  These  probabilities  are 
necessary  for  every  phase,  tactic,  and  weapon  combination.  While  this  modeling 
approach  provides  a  more  realistic  representation,  it  is  not  clear  where  an  analyst  can  find 
all  of  these  probabilities. 

The  maximum  potential  number  of  both  answered  and  unanswered  volleys  is 
computed  for  all  possible  shooter-target  pairs.  Total  volleys  are  the  number  of 
opportunities  (taken  or  not)  a  given  aircraft/weapon  pair  has  to  fire  a  volley  against 
aircraft  in  an  opposing  flight.  The  maximum  potential  number  of  volleys  is  calculated  for 
every  weapon  against  every  potential  opposing  flight.  Only  weapons  that  are  employable 
in  the  specific  phase  are  considered  in  the  calculation.  The  maximum  number  of 
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unanswered  volleys  is  drawn  from  a  triangular  distribution  with  the  minimum,  mode,  and 
maximum  input  by  the  user.  This  data  is  designed  to  be  “...derived  directly  from  higher 
resolution  models  that  represent  the  specific  characteristics  of  and  interactions  between 
the  aircraft  and  weapons  involved  in  the  engagement”  [28:1 1].  The  triangular 
distribution  was  chosen  to  make  data  development  easier.  This  value  is  entered  in  the 
A2A.DAT  shown  in  Appendix  C  file  by  phase,  weapon,  tactic,  and  target  class.  Target 
class  is  a  particular  category  where  an  opponent  is  categorized,  such  as  fighter,  bomber, 
or  other.  Fractional  draws  from  the  triangular  distribution  are  randomly  rounded  up  or 
down  to  form  an  integer  value. 

The  developers  of  STORM  underwent  an  extensive  calibration  process  in  an 
attempt  to  adjust  the  input  variables  of  the  STORM  adjudicator  to  that  of  BRAWLER. 
During  this  process,  they  discovered  that  a  triangular  distribution  may  not  be  appropriate 
because  the  necessary  parameters  of  minimum,  mode,  and  maximum  are  not  readily 
collected  from  BRAWLER.  The  problem  lay  in  the  oversimplification  forced  by  the 
triangular  distribution  and  the  fact  that  the  computed  mode  was  really  the  mean  volleys 
taken  in  BRAWLER.  The  problem  is  illustrated  by  Figure  5  [29:18]. 
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Possible  Volley  Distributions 


Figure  5.  Gamma  and  Triangular  Distributions 

Since  the  actual  shot  and  volley  distribution  more  closely  represents  a  gamma 
distribution  like  the  one  shown  above  rather  than  a  triangular  one,  the  average  number  of 
shots  produced  was  consistently  too  high.  The  maximum,  therefore,  was  tuned  to  give 
the  correct  average.  This  causes  the  prototype  algorithm  to  lose  some  of  its  ability  to 
adequately  represent  the  rarely  occurring  upper  end  of  the  volley  spectrum.  The 
developers  have  chosen  to  remedy  the  problem  in  the  production  version  of  STORM  by 
allowing  the  user  to  define  the  type  of  distribution  used  in  determining  volley 
computation  [29 : 1 7] . 

When  a  number  is  drawn  from  one  of  these  continuous  distributions,  the 
developers  of  STORM  have  chosen  the  following  algorithm  to  randomly  round  all 
decimal  values  higher  or  lower: 
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(23) 


Let  D  =  Random  number  drawn  from  appropriate 

distribution  given  parameters  set  by  the  user 

Let  I  =  Truncate  (D) 

Let  D  =  D  - 1 

IfRJV.[0,l]  <  D:  Add  1  to  I 

Return  I 

The  algorithm  then  computes  the  number  of  unanswered  volleys  that  the 
particular  side  shoots.  This  is  computed  by  a  series  of  random  sample  draws  from  a 
binomial  distribution.  The  probability  of  success  is  the  same  probability  used  above,  i.e. 
the  probability  that  the  particular  side  shoots  the  unanswered  volley.  For  the  binomial 
draw,  the  number  of  trials  from  which  the  sample  is  taken  is  the  “max  unanswered 
volley”  calculation  described  above. 

3.3.4  Volley  Adjudication 

The  volley  is  initialized  by  a  computation  of  the  force  ratio  of  the  opposing  flight 
groups.  Each  flight  group  score  is  computed  by  summing  the  scores  of  the  individual 
elements.  The  scores  of  the  individual  elements  are  found  by  adding  together  the  user- 
input  element  configuration  scores  by  type  aircraft  and  munition.  The  force  ratio  is 
computed  using  the  following  equations: 

_  _  BlueConfigurationScore  (24) 

BlueForceRatio  = - 

RedConfigurationScore 

_  .  RedConfigurationScore 

RedForceRatio  = - — - 

BlueConfigurationScore 
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The  flight  group  command  and  control  capability  is  also  initialized  at  the 
beginning  of  the  volley  sequence.  The  user  has  the  ability  to  enter  a  flight  group 
command  and  control  capability  value,  expressed  as  a  percentage.  This  is  to  simulate  the 
“...degree  of  inter-flight  coordination  reflecting  the  contribution  of  external  systems 
(AW ACS,  GCI,  etc.)  as  well  as  onboard  aircraft  systems  (datalink,  voice,  etc.)  and  pilot 
training  [28:10].”  This  parameter  may  be  dynamically  adjusted  according  to  user-defined 
rules  based  on  the  availability  and  condition  of  relevant  systems.  Using  this  user-defined 
percentage,  a  random  number  draw  determines  if  a  flight  group  has  command  and  control 
and  signals  a  “flag”  to  the  rest  of  the  algorithm.  If  the  random  number  is  less  than  the 
command  and  control  value,  then  the  flight  group  does  not  have  command  and  control, 
otherwise,  it  does. 

The  presence  or  absence  of  command  and  control  affects  the  perceived  number  of 
enemy  aircraft.  If  a  flight  group  has  command  and  control,  the  number  of  perceived 
enemy  aircraft  is  equal  to  the  actual  number  of  aircraft.  If  the  flight  group  does  not  have 
command  and  control,  the  perceived  number  of  enemy  aircraft  is  drawn  from  a  normal 
distribution  with  the  following  parameters: 

p  =  Actual  Number  of  Enemy  Aircraft  (25) 

a  =  (1  -  FGCC)  *  p 

where, 

FGCC  =  The  flight  group  command  and  control  specified  by  the  user. 
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As  seen  in  Figure  6,  as  the  number  of  aircraft  increases  and  the  FGCC  decreases 
from  perfect  command  and  control  {FGCC  =  1)  to  no  command  and  control  {FGCC  =  0), 
the  standard  deviation  increases  for  each  scenario.  This  is  consistent  with  reality.  As  the 
number  of  red  aircraft  increases  and  the  blue  flight  group  command  and  control 
decreases,  the  blue  aircraft  are  more  likely  to  incorrectly  guess  the  number  of  red  aircraft. 

The  perceived  number  of  aircraft  is  randomly  rounded  up  or  down  to  the  nearest 
whole  number  using  the  same  algorithm  (23)  that  rounds  the  triangular  distribution.  The 
same  concern  applies  that  was  discussed  previously  about  using  a  uniform  rounding  rule 
to  round  the  numbers  up  or  down. 

Also  initialized,  is  the  desire  to  escape  of  the  individual  flight  groups.  If  the  force 
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Actual  Number  of  Enemy  Aircraft 


Figure  6.  Standard  Deviation  of  Perceived  Number  of  Aircraft 
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ratio  drops  below  the  user-input  force  ratio  threshold  (example  ofESCAPE.dat  file 
shown  in  Appendix  D),  the  individual  element  will  desire  to  escape  and  will  continue  to 
desire  to  escape  through  the  remaining  phases.  Also,  an  element’s  desire  to  escape  is 
based  on  the  number  of  weapons  that  the  element  has.  If  the  total  air-to-air  configuration 
score  for  that  element  drops  below  a  specified  threshold,  that  element  will  desire  to 
escape.  The  force  ratio  is  computed  using  (24).  Notice  that  it  uses  the  actual  score  of  the 
enemy  aircraft  and  not  the  perceived  score.  This  raises  concerns  that  will  be  discussed  in 
the  next  section. 

Once  all  of  the  initialization  processes  are  completed,  the  volley  adjudication 
follows  five  main  steps  -  targeting  decisions,  weapon  selection  and  expenditure,  escape 
decision  and  result,  and  attrition  calculation. 

3.3, 4.1  Targeting  Decision 

The  opponent  “perceived  score”  is  also  initialized  at  this  stage.  The  perceived 
score  is  computed  as: 

Perceived  Score  =  Tot  Number  of  Aircraft  *  (26) 

Max(Configuration  Score,  Minimum  Perceived  Score ) 

The  minimum  perceived  score  is  a  user-specified  input  in  A2A.dat  (see  Appendix 

C).  As  shown  from  (26),  the  maximum  of  either  the  configuration  score  or  the  minimum 
perceived  score  is  used.  Notice  that  the  perceived  score  is  based  on  the  actual  total 
number  of  aircraft.  It  seems  that  the  perceived  number  of  aircraft  should  be  based  on  the 
perceived  number  of  aircraft  using  a  distribution  draw  discussed  previously  in  (25)  rather 
than  the  actual  number  of  aircraft.  This  score  is  attempting  to  quantify  one  side’s 
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perception  of  the  opponent’s  capabilities.  Therefore,  this  score  should  then  be  based  on 
the  number  of  aircraft  that  the  side  believes  the  opponent  has. 

During  the  targeting  decision  portion  of  the  algorithm,  each  flight  group’s 
elements  are  grouped  against  flights  of  the  opponent.  The  aircraft  of  a  particular  side  are 
first  sorted  by  the  aircraft-munition  configuration  score  from  highest  to  lowest,  and  the 
opponent’s  side  is  sorted  by  perceived  flight  score.  The  force  ratio  computation 
determines  the  number  of  elements  to  target  a  particular  flight  and  is  given  by: 

_  _  .  BlueConfigurationScore  (27) 

BlueForceRatio  = - 

RedPerceivedScore 

_  _  .  RedConfigurationScore 

ReaForceRatio  = - 

BluePerceivedScore 

The  perceived  score  is  computed  using  (26).  Blue  elements  are  allocated  to  Red 
flights  until  this  force  ratio  is  greater  than  1 .  This  process  allocates  friendly  elements  to 
enemy  flights  and  is  performed  by  both  sides.  An  element  can  be  assigned  to  one  and 
only  one  enemy  flight  within  a  single  volley.  Once  the  force  ratio  has  been  exceeded  for 
all  opposing  flights,  the  remaining  friendly  flights  do  not  target  anyone.  Over  multiple 
volleys,  a  single  element  may  engage  multiple  flights. 

Notice  that  the  force  ratio  used  in  computing  the  desired  force  ratio  is  based  on 
the  opponent’s  perceived  platform  configuration  score.  It  seems  that  the  equation  for  the 
ratio  in  (24)  to  compute  the  desire  to  escape  should  be  computed  in  the  very  same  way.  It 
should  also  be  based  on  the  perceived  score  of  the  opponent.  Notice  that  (27)  contains 
the  perceived  score  in  the  denominator  while  (24)  contains  the  true  configuration  score. 
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The  desire  to  escape  should  also  be  determined  based  on  the  number  of  aircraft  the 
friendly  side  perceives  the  opponent  has. 

As  the  algorithm  is  designed  currently,  the  enemy  flights  are  sorted  highest  to 
lowest  and  the  friendly  elements  are  sorted  highest  to  lowest.  The  greedy  algorithm 
described  above  allocates  the  friendly  elements  to  the  flights  with  the  highest  perceived 
air-to-air  configuration  score  until  the  desired  force  ratio  is  reached.  This  may  work  best 
for  most  situations,  but  it  does  not  take  into  account  the  presence  of  high  value  assets 
(HVA).  A  possible  HVA  scenario  may  include  a  flight  of  MIG-29s  engaging  a  flight  of 
F-15s  escorting  B-ls  in  enemy  airspace  enroute  to  bombing  a  target.  Depending  on  the 
existing  force  ratio,  the  MIG-29s  in  reality  may  place  a  high  priority  on  targeting  the  B- 
1  s,  bypassing  the  F-l  5s.  This  is  just  one  example  in  reality  where  a  friendly  element  may 
chose  to  target  a  known  HVA  and  try  to  circumvent  the  fighters  that  may  have  a  high 
configuration  score.  This  is  currently  not  modeled  in  the  algorithm.  An  HVA  may  have 
a  low  aircraft  configuration  score  and  would  therefore  be  lower  in  priority  to  target  by  the 
opponent. 

3.3.4.2  Weapon  Selection  and  Expenditure 

Weapon  selection  is  also  accomplished  for  every  element  against  the  opposing 
flights  with  munitions  preferences  specified  by  the  user  according  to  phase.  The 
algorithm  looks  for  the  first  item  on  the  list  in  order  of  preference  that  does  not  have  a 
probability  of  weapons  effectiveness  (Pwe  -  also  a  user-input  value)  equal  to  zero  and 
more  than  zero  volleys  remaining. 
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The  maximum  number  of  weapons  and  targets  is  then  computed  as  a  function  of 
user-input  values.  A  salvo  is  a  set  number  of  weapons  to  fire  at  each  aircraft.  Each  type 
of  aircraft  may  have  the  ability  to  multi-target  and  the  potential  to  fire  salvos  at  up  to  that 
number  of  different  enemy  aircraft.  The  number  of  salvos  is  computed  as  follows: 

_  ,  mxn(NumberofAirMunitions,MaxTgts*  MaxWeaponsPerTgtPerVolley)  (28) 

Salvos  = - — — — - „  ——  „ - 

Max  WeaponsPerTgtPerVolley 

where, 

MaxTgts  =  NumberlnElement  *  MaxTgtsPerVolley 

The  total  shots  within  a  volley  are  limited  by  munition  availability  as  well. 

Salvos  are  allocated  to  the  enemy  flights  starting  from  the  aircraft  with  the  greatest  air-to- 
air  ability  (determined  from  the  aircraft-munition  configuration  score)  [28:12]. 

The  shots  are  then  allocated  individually  to  the  target  aircraft.  At  this  point,  the 
element  command  and  control  is  significant.  The  allocation  of  shots  can  range  from  an 
optimal  (i.  e. ,  even)  distribution  to  a  random  distribution  of  salvos  to  target  aircraft 
depending  upon  the  command  and  control  of  the  elements  involved.  The  allocation  of 
random  shots  from  a  lack  of  command  and  control  can  result  in  some  enemy  aircraft 
being  double  targeted. 

3.3.4.3  Escape  Decision  and  Result 

The  desire  to  escape  calculation  is  accomplished  in  the  volley  initialization 
process  discussed  previously.  The  actual  ability  for  an  element  to  escape  comes  before 
the  volley  exchange  and  is  based  on  the  current  phase,  disparity  in  force  ratio,  and  the 
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types  of  opponents  engaged.  First,  the  probability  of  escape  is  calculated  using  user- 
input  values. 


Pr {Escape)  = 


1 


1  +  a  •  exp(-/?  •  ForceRatio  ) 


Where, 


P  = 


(log(  ^-log^) 

_ X2 


x2-x, 


a  =  exp  (/?  •  X,  +  log(— — — 

■M 


(29) 


(30) 


(31) 

(32) 


Force  Ratio  (Flight  Group  1)  =  (Flight  Group  1  Score)  /  (Flight  Group  2  Score) 
or 

Force  Ratio  (Flight  Group  2)  =  (Flight  Group  2  Score)  /  (Flight  Group  1  Score) 
and, 

Xi  =  Low  Force  Ratio 

Y]  =  Low  Probability  of  Escape 

X2  =  High  Force  Ratio 

Y2  =  High  Probability  of  Escape 

Xi,  Y],  X2,  Y2  are  input  by  the  user  according  to  phase  and  category  of  aircraft  as 
shown  in  Attachment  D.  This  distribution  works  well  to  accommodate  a  large  variety  of 
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Figure  8.  Increasing  Beta  in  Escape  Function 


For  example,  consider  the  case  of  a  bomber  trying  to  escape  from  enemy  fighters. 
Say  that  the  user  inputs  the  following  combinations  to  describe  a  fighter  against  a 
bomber: 

•  For  a  low  force  ratio  of  1  blue  to  5  reds  (0.2),  there  is  the  probability  of  escape 
of  0.20 

•  For  a  high  force  ratio  of  5  blues  to  1  red  (5),  there  is  a  probability  of  escape  of 
0.50 

After  these  values  are  input  into  (30)  and  (31),  Figure  9  shows  the  resulting 
distribution  for  (29).  If  the  actual  force  ratio  turned  out  to  be  5  blues  to  1  red,  the  bomber 

would  have  around  an  1 8%  chance  of  escaping.  This  function  can  be  tailored  to  meet  the 
likelihood  of  an  escape  for  any  aircraft  combination  that  participates  in  the  engagement. 


Force  Ratio 

Figure  9.  Example  Case  for  Escape  of  Bomber  vs  Fighter 
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For  another  example,  consider  a  fighter  escaping  from  a  less  capable  fighter,  such 
as  a  F-15A  from  a  MIG-21.  Say  that  the  user  inputs  the  following  combinations  to 
describe  a  fighter  against  another  fighter: 

For  a  low  force  ratio  of  1  blue  to  5  reds  (0.2),  there  is  the  probability  of  escape 
of  0.40 

For  a  high  force  ratio  of  5  blues  to  1  red  (5),  there  is  a  probability  of  escape  of 
0.70 

This  would  result  in  the  function  shown  in  Figure  10  describing  the  F-15A’s 
ability  to  escape  for  a  variety  of  force  ratios.  If  the  actual  force  ratio  turned  out  to  be  2 
blues  to  1  red,  the  F-15A  has  at  least  a  50%  chance  of  escaping. 


0  1  2  3  4  5  6 

Force  Ratio 


Figure  10.  Example  Case  for  Escape  Bomber  vs  Fighter 
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Aircraft  can  be  grouped  according  to  their  ability  to  escape.  The  probability  of 
escape  is  scaled  by  the  fraction  of  the  phase  that  the  element  wanted  to  escape. 

Also,  a  bias  can  be  input  by  the  user  to  account  for  the  timing  of  the  escape 
opportunities.  For  instance,  all  escape  opportunities  may  occur  before  the  first  or  last 
volley  or  may  be  equally  distributed  among  all  volleys.  In  highly  advantageous  force 
ratio  conditions,  even  the  slowest  most  vulnerable  aircraft  may  be  able  to  escape  easily. 
Conversely,  when  heavily  outnumbered  by  highly  capable  systems,  even  an  advanced 
aircraft  may  have  difficulty  escaping  in  a  WVR  engagement.  A  probability  of 
successfully  egressing  the  engagement  is  selected  based  upon  these  factors  [28:12]. 

3.3.4A  Attrition  Calculation 

After  shots  have  been  allocated  and  munitions  are  consumed,  attrition  is 
calculated  based  upon  the  random  draws  against  the  user-input  probability  of  weapon 
effectiveness  (Pwe)-  Pwe  represents  the  probability  of  kill  given  a  firing  at  an  alert  enemy 
target.  The  Pwe  (shown  in  Appendix  C)  is  a  function  of  the  shooter,  weapon,  shooter’s 
tactics,  target,  and  target’s  tactics  involved.  This  data  value  requirement  was  designed  to 
be  gathered  from  higher-resolution  models.  An  aircraft  is  killed  if  a  random  number 
from  a  U(0,1)  >  (1  -  Pwe)- 

Once  that  an  aircraft  is  killed,  STORM  accurately  accounts  for  the  lost  munitions 
associated  with  that  killed  aircraft.  The  algorithm  specifically  subtracts  those  munitions 
as  lost  in  the  engagement. 
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Possible  problems  may  arise  in  possible  “double  dipping”  between  the  Pwe  and  the 
probability  of  an  unanswered  volley.  As  the  data  definitions  and  values  are  determined, 
either  by  testing,  SMEs,  or  calibration  with  lower  models,  the  definitions  and  differences 
between  these  two  data  requirements  should  be  understood  by  all  parties  involved. 
Otherwise,  when  determining  the  probability  of  an  unanswered  volley,  there  may  be 
confusion  regarding  the  probability  that  the  weapon  shoots  down  the  target  first. 

During  the  developers  calibration  efforts,  the  Pwe  was  linked  to  the  Pweaponeffect 
from  BRAWLER.  These  values  were  modified,  however,  based  on  the  number  of  “Fuze 
on  Dead  Target”  missiles.  Dead  Target  Fuzings  would  most  likely  be  caused  by  two 
separate  aircraft  shooting  at  the  same  target,  which  would  be  less  likely  given  sufficient 
Command  and  Control.  Since  STORM  models  Command  and  Control  separately,  and 
Command  and  Control  affects  the  number  of  targets  seen  and  target  apportionment,  using 
a  Pweaponeffect  which  was  already  attenuated  by  “Fuze  on  Dead  Targets”  would  double 
count  this  effect  and  result  in  a  lower  Pwe  than  originally  intended.  Therefore,  the 
developers  corrected  this  problem  with  this  equation  [29: 16]: 

STORM  Pwe  =  Kills/(Shots  -  Fuzes  on  Dead  Targets )  (33) 

An  experiment  was  conducted  to  verify  the  reasonableness  of  the  attrition  for  various 
engagement  scenarios.  A  flight  of  F-15Cs  with  tolerant  tactics  engage  MIG-29s  also  with 
tolerant  tactics.  The  F-15Cs  carried  the  AIM-120  and  the  AIM-9X.  The  MIG-29s 
carried  AA-10  and  AA-1 1  missiles.  Average  kills  over  100  replications  were  measured 
for  the  following  F-15C  versus  MIG-29  engagements:  12vl2,  2vl2,  2v4,  2v8,  4v4,  and 
8v8.  Figure  1 1  shows  the  results  from  each  of  these  engagements.  The  exchange  ratio  at 
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Figure  11.  Results  of  Various  Force  Ratio  Engagements 


the  top  of  the  graph  is  an  indicator  to  the  relative  performance  of  one  side  to  the  other. 
The  exchange  ratio  is  equal  to  the  number  of  blue  kills  (red  dead)  divided  by  the  number 
of  red  kills  (blue  dead).  The  model  performed  as  expected  given  the  input  parameters  fed 
into  it.  Due  to  the  better  input  parameters  given  to  the  F-l  5C,  it  outperforms  the  MIG-29. 
Even  in  the  2vl2  case,  the  F-15s  kill  5  MIGs  before  the  F-15s  are  actually  destroyed 
themselves.  The  closest  matchup  seems  to  be  the  8v8  case  where  each  side  destroys 
approximately  3  adversaries. 


3.4  Summary 

This  chapter  has  presented  an  overview  of  the  STORM  air-to-air  adjudicator 
methodology.  Overall,  STORM  has  captured  most  of  the  critical  elements  of  the  air-to- 
air  engagement  process.  Table  3  shows  how  elements  of  the  air-to-air  combat 
environment  are  modeled  either  explicitly  or  implicitly. 
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Modeled 

Explicitlv 

Modeled 

Imnlicitlv 

M  issions 

Aircraft  &  Org 
Representation 

V 

Command  &  Control 

V 

Spatial  Layout 

V 

Environment 

Munitions  & 

Expenditures 

V 

Tactics  &  Maneuvering 

V 

Sensor  Performance 

V  ] 

Detection 

V 

Identification 

V  I 

Electronic  W arfare 

Table  3.  Elements  of  Air-to-air  Combat  Modeled  by  the  STORM  Adjudicator 

There  were  significant  methodology  improvements  that  model  attrition  and 
munitions  consumption  with  a  higher  degree  of  fidelity  than  THUNDER. 

Some  of  the  aspects  of  air-to-air  combat  STORM  models  particularly  well  are: 

•  The  phase  concept  improves  overall  transparency  and  captures  most  of  the 
issues  in  the  air-to-air  combat  environment  appropriate  to  a  campaign  level 
model. 

•  The  volley  concept  accurately  represents  munitions  consumption. 
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•  The  probability  of  an  unanswered  volley  realistically  accounts  for  range 
advantages  and  can  be  tied  to  aircraft  platform. 

•  When  a  random  number  is  drawn  from  a  normal  distribution  to  define  the 
number  of  perceived  aircraft,  the  parameters  for  the  mean  and  standard 
deviation  of  the  perceived  number  of  aircraft  is  accurately  computed. 

•  The  ability  to  escape  is  based  on  an  innovative  function  that  can  be  shaped  by 
the  user-input  values. 

•  Pwe  can  be  tied  to  the  shooting  aircraft  platform. 

•  The  tracking  of  lost  munitions  of  killed  aircraft  is  accurately  accounted  for. 

Some  areas  of  improvement  and  concerns  are: 

•  Definition  of  phases  should  be  determined  and  agreed  upon  by  the  users  of 
STORM. 

•  Certain  issues  of  time  and  distance  should  be  modeled  either  inside  or  outside 
the  adjudicator  (e.g.  fuel  constraints  and  position  relative  to  the  FLOT  affect 
engagement  strategies) 

•  The  perceived  score  of  an  opponent  is  based  on  the  actual  total  number  of 
aircraft  instead  of  the  perceived  number  of  aircraft.  It  should  be  based  on 
what  the  friendly  believes  the  opponent  has. 

•  There  is  some  concern  that  Pwe  there  would  be  “double  dipping”  with  the 
probability  of  an  unanswered  volley  unless  the  definitions  are  clearly 
understood  by  users. 
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•  The  targeting  decision  does  not  include  circumstances  involving  HVAs.  An 
HVA  may  have  a  low  aircraft  configuration  score  and  would  therefore  be 
lower  in  priority  to  target  by  the  opponent. 

The  methodology  presented  in  this  chapter  assumed  that  the  data  was  available  for 
modeling.  The  issue  of  data  source  and  availability  will  be  addressed  in  the  next  chapter 
where  the  weaknesses  of  THUNDER  and  STORM  will  be  compared. 
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4.  Comparison  of  the  THUNDER  and  STORM  Models 


4.1  Introduction 

The  limitations  of  THUNDER  have  circulated  within  the  user  community  for 
many  years.  Many  complaints  have  merit,  but  others  can  be  traced  to  misunderstandings 
due  to  a  lack  of  knowledge  of  either  the  assumptions  of  THUNDER  itself  or  how  it 
operates.  The  developers  of  STORM  have  documented  a  subset  of  the  user  concerns. 
Phase  II  of  this  report  occurred  in  September,  1998  [27].  The  following  is  a  sample  of 
some  of  the  complaints  of  the  air-to-air  adjudicator: 

•  “...THUNDER  automatically  expends  missiles  when  an  engagement  is 
initiated  based  on  the  user-defined  number  of  launches  per  engagement. 

Some  of  these  are  not  adjudicated.. .while  others  are  wasted  on  ‘many-on-few’ 
engagements.” 

•  “...Weapon  expenditures,  stealth  detection,  command  and  control  are  all 
subjective  and  subject  obscure  input  data.  However,  when  calibrated  properly 
with  a  model  like  BRAWLER,  the  results  could  not  be  easily  dismissed. 
Probably  the  weakest  area  was  the  determination  of  when  and  where  an 
engagement  occurs.  Both  CAP  and  INTERCEPT  methodologies  are  very 
weak.  Weapon  expenditures  are  probably  the  next  biggest  problem.” 

•  “...it  is  difficult  to  link  back  to  engagement  level  models  and  has  limited 
utility  for  munitions  expenditure  studies.” 
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•  “It  seems  to  work,  but  is  not  very  well  understood  and  somewhat  mystical. 
Data  inputs  of  relative  range  advantage,  friendly  fighter  lethality,  value  of 
friendly  loss,  etc.  are  more  subjective  than  objective,  and  their  effects  on  air 
combat  outcomes  are  not  very  straight  forward.” 

THUNDER  and  STORM  differ  in  their  modeling  approaches.  THUNDER  is  a 
time-stepped,  probabilistic  representation  of  interactions  occurring  in  air-to-air  combat. 
STORM  is  a  heuristic,  event-oriented  approach  to  modeling  air-to-air  engagements. 
STORM’s  methodology  was  chosen  to  improve  transparency  to  users  and  to  increase  the 
resolution  of  campaign  level  air-to-air  combat  beyond  THUNDER.  This  will  improve 
the  accounting  of  attrition  and  munitions  expenditures  in  the  air-to-air  combat 
environment.  Due  to  these  different  modeling  approaches,  it  makes  it  more  challenging 
to  compare/contrast  specific  modeling  issues  across  both  models.  Also,  as  mentioned 
previously,  Denhard  has  identified  four  stated  criticisms  of  the  THUNDER  air-to-air 
submodel:  1)  single  shot  probability  of  kill  and  range  advantage  issues,  2)  the  number  of 
weapons  fired  per  engagement  by  an  aircraft  is  the  same  for  all  air-to-air  engagements 
regardless  of  type  of  opponent  aircraft  faced,  3)  a  SHOOT-SHOOT  firing  doctrine 
instead  of  the  more  typical  USAF  SHOOT-LOOK-SHOOT  firing  doctrine,  and  4)  does 
not  allow  for  aircraft  to  disengage  before  weapons  release  [8:79].  These  concerns  will  be 
addressed  in  the  following  sections. 

4.2  Probability  of  Kill  Calculation 

The  single  shot  probability  of  kill  employed  by  THUNDER  considers  only  the 
firing  platform/weapon  combination  versus  opponent  platform  and  not  the  opponent’s 
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weapon.  Consider  the  example  of  an  F-15C  armed  with  an  AIM- 120  as  its  primary 
weapon  and  the  AIM-9  as  the  secondary  weapon.  It  is  engaged  by  a  threat  aircraft  armed 
with  a  BVR  missile  that  is  less  capable  than  the  AIM- 120  but  has  a  greater  range  than  the 
AIM-9.  In  real  life,  the  AIM-120  is  likely  to  dominate  the  majority  of  engagements. 

This  ends  up  negating  the  enemy’s  range  advantage  over  the  AIM-9.  However,  if  the 
F-15C  expends  all  AIM- 120s  during  previous  engagements,  the  range  advantage  of  the 
opponent’s  weapon  over  the  AIM-9  is  not  reflected  because  the  probability  of  kill  does 
not  take  into  account  the  opponent’s  weapon.  In  other  words,  you  cannot  show  a  realistic 
difference  between  using  it  as  a  secondary  munition  (after  the  AIM- 120,  where  the 
opponent’s  weapon  does  not  cause  a  problem)  versus  using  it  as  a  primary  munition 
(where  the  opponent’s  weapon  would  cause  problems). 

STORM  solves  this  problem  with  the  concept  of  unanswered  volleys.  Recall  that 
the  possibility  of  an  unanswered  volley  takes  into  account  a  relative  range  advantage. 

This  probability  of  an  unanswered  volley  is  input  by  the  user  according  to  phase,  friendly 
weapon,  friendly  tactics,  and  opponent  weapon.  Therefore,  since  the  probability  is  based 
partly  on  the  opponent’s  weapon  type,  this  value  would  show  the  advantage  of  the 
opponent’s  weapon  over,  say,  the  AIM-9,  given  that  the  AIM- 120  was  already  expended. 

4.3  Munitions  Consumption 

THUNDER’S  air-to-air  model  has  been  useful  but  “...suffers  from  limiting 
assumptions  that  oversimplify  the  problem”  [28:7].  THUNDER  is  good  at  representing 
attrition  but  lacks  the  ability  to  accurately  represent  munitions  consumption.  One 
limiting  assumption  is  the  requirement  that  the  user  input  the  number  and  type  of 
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weapons  fired  for  any  engagement.  The  user  inputs  this  value  according  to  the 
configuration  of  the  aircraft  and  its  mission.  This  limiting  assumption  is  particularly 
troublesome  where  there  are  disparate  force  numbers  between  friendly  and  opponent 
aircraft. 

STORM  overcomes  this  problem  through  the  use  of  the  volley  concept.  The  user 
specifies  the  maximum  number  of  weapons  to  fire  per  volley  by  aircraft  and  munition 
type.  Notice  that  this  is  “per  volley”  and  not  “per  engagement.”  This  allows  for  the  shots 
to  be  fired  and  an  assessment  to  determine  if  the  aircraft  were  destroyed.  This  prevents 
over-shooting  of  opponent  aircraft  and  more  accurately  model  munitions  expenditures.  If 
a  flight  group  does  not  have  accurate  command  and  control,  there  may  be  over-shooting 
of  opponent  aircraft,  but  this  is  intentionally  modeled  to  account  for  this  possible  real- 
world  situation. 

An  experiment  was  conducted  on  the  STORM  air-to-air  adjudicator  to  compute 
the  number  of  munitions  expended  across  a  variety  of  force  ratio  scenarios.  All  scenarios 
involve  F-l  5Cs  versus  MIG-29s,  both  fully  loaded  with  “tolerant”  tactics  strategy.  100 
replications  were  conducted  for  each  scenario.  Figure  12  displays  the  average  number  of 
weapons  expended  in  STORM  for  each  scenario  by  munition  type.  The  figure  also 
shows  the  number  of  weapons  expended  in  THUNDER.  The  solid  line  shows  the  total 
number  of  munitions  expended  in  THUNDER  for  the  red  forces  (AA-lOs  and  AA-1  Is), 
and  the  dotted  line  shows  the  total  number  of  munitions  expended  in  THUNDER  for  the 
blue  forces  (AIM- 120s  and  AIM-9Xs).  This  assumes  input  into  THUNDER  of  2  shots 
per  engagement. 


69 


BAA-10 

□  AA-11 

BRed-THUNDER 

HAIM-9X 

HAIM-120 

BBIue-THUNDER 

25  A 


12v12  12v2  4v2  8v2  4v4  8v8 


Figure  12.  STORM  vs  THUNDER  Munition  Expenditures 

Notice  in  the  figure  that  in  a  scenario  with  disparate  force  ratios,  such  as  2vl2  (2 
F-15s  versus  12  MIG-29s),  that  THUNDER  calculates  the  blue  side  taking  4  shots  and 
red  side  taking  24  shots.  THUNDER  has  been  criticized  for  not  modeling  munitions 
expenditures  correctly  in  situations  with  a  great  disparity  in  force  ratios  -  24  red  shots 
against  only  two  blue  aircraft  is  clearly  overkill.  In  the  STORM  algorithm,  the  amount  of 
munitions  expended  from  the  MIG-29s  is  more  reasonable  (Red:  5  AA-lOs  and  1  AA- 
1 1 ;  Blue:  7  AIM-120s  and  1  AIM-9X).  In  scenarios  where  the  forces  are  smaller  and 
more  equally  matched,  such  as  a  4v4  case,  the  THUNDER  modeling  approach  is 
acceptable.  This  experiment  demonstrates  that  STORM  corrects  THUNDER’S  problem 
of  inadequately  modeling  disparate  force  ratio  scenarios. 
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4.4  Firing  Doctrine 


THUNDER  treats  a  multiple  missile  salvo  like  a  SHOOT-SHOOT  firing  strategy 
(which  drives  down  the  average  single  missile  effectiveness)  while  BRAWLER  treats  a 
multiple  missile  salvo  as  SHOOT-LOOK-SHOOT  firing  tactic  [8:79].  STORM 
disaggregates  the  engagements  into  a  series  of  volleys  that  make  up  a  particular 
engagement.  This  allows  a  more  realistic  representation  of  the  Air  Force  doctrine  of  a 
SHOOT-LOOK-SHOOT  firing  strategy  instead  of  a  SHOOT-SHOOT  strategy. 

4.5  Escape  Before  Weapons  Release 

When  an  engagement  occurs  in  THUNDER,  weapons  are  always  fired.  The 
model  does  not  allow  for  aircraft  to  disengage  before  weapons  release.  In  the  STORM 
adjudicator,  this  is  corrected  when  the  probability  of  escape  calculation  is  performed. 

This  calculation  occurs  before  and  after  and  in-between  the  unanswered  and  answered 
volley  logic.  This  allows  the  aircraft  to  escape  even  before  there  is  a  single  shot  fired. 

There  are  cases  where  one  aircraft  might  fire  its  weapons,  temporarily  escape,  re¬ 
engage,  fire  its  weapons,  temporarily  escape,  and  continue  this  cycling  strategy. 
Currently,  the  STORM  adjudicator  models  this  cycling  strategy  with  the  tactics  concept, 
but  this  only  occurs  within  the  adjudicator  itself.  A  situation  where  you  have  such  a 
cycling  strategy  would  be  modeled  with  an  averse  tactic  strategy.  The  result  is  that  firing 
a  missile  at  an  averse  aircraft  would  have  a  lower  probability  of  success  than  a  missile 
fired  at  a  tolerant  aircraft.  Besides  modeling  with  a  tactics  strategy,  there  are  no  other 
opportunities  for  modeling  such  a  cycling  strategy.  When  the  algorithm  is  placed  within 
STORM,  the  adjudicator  should  allow  aircraft  to  escape  completely,  end  the  engagement, 
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and  then  re-engage  to  start  a  new  engagement.  In  other  words,  the  adjudicator  could  loop 
back  to  itself. 

4.6  Data  Requirements 

The  developers  of  STORM  categorized  some  of  the  concerns  about  data 
requirements  into  several  areas.  First,  there  should  be  links  to  other  models.  Campaign 
models  should  be  supported  by  data  from  engagement  or  mission  level  models. 
Automating  these  links  and  designing  compatible  assumptions  in  the  campaign  model 
methodology  ensures  an  adequate  data  source.  Many  times  this  effort  is  complicated  by 
the  need  for  thousands  of  engagement  model  runs.  Another  goal  is  to  provide  for  links  to 
data  repositories.  Existing  data  repositories  could  be  a  source  from  which  to  gather  input 
requirements.  Many  times  this  effort  is  complicated  by  the  need  for  further  “data 
massaging”  in  order  to  match  the  needed  data  requirement.  Another  goal  is  to  provide  for 
links  to  real  world  systems.  Real  world  systems  can  be  a  source  of  timely  data  that  may 
not  have  been  considered  a  threat  until  a  situation  arises.  The  data  requirements  that  are 
required  for  running  STORM  will  be  evaluated  in  light  of  these  goals. 

In  association  with  ASC/ENMM,  Table  4  (located  at  the  end  of  this  chapter)  has 
been  created  to  evaluate  the  various  data  requirements  required  in  the  STORM  air-to-air 
algorithm.  The  categories  of  interest  are  the  possible  sources  of  locating  such  data,  the 
possible  security  level,  whether  that  data  would  be  acquired  from  another  model,  and 
other  comments  or  concerns.  Most  of  the  concerns  listed  in  the  comments  heading  have 
been  addressed  in  the  previous  chapter. 
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Because  the  adjudicator  is  still  in  the  prototype  stage,  some  of  the  parameters  are 
listed  as  user  input  but  will  be  dynamic  in  nature  when  the  actual  model  is  implemented. 
Such  things  as  ATOs,  AEW  coverage,  and  various  mission  types,  done  outside  the 
adjudicator,  will  require  that  the  input  parameters  be  dynamic.  Therefore,  a  caveat  to  this 
evaluation  is  that  some  of  the  data,  such  as  command  and  control,  will  be  affected  by 
factors  outside  the  adjudicator. 

The  overall  evaluation  of  the  data  requirements  reveals  that  many  of  the  data 
requirements  appear  to  be  very  subjective  at  this  point.  Much  of  the  data  will  probably 
be  gathered  from  consultations  with  a  combination  of  SMEs  and  intelligence  personnel. 
Some  of  the  data  will  find  its  origin  in  such  models  as  BRAWLER,  but  will  require 
extensive  “data  massaging”  to  acquire  correct  results.  It  is  highly  recommended  that  a 
significant  investment  of  time  and  resources  be  put  into  the  development  of  such  data. 
The  adjudicator’s  ability  to  match  reality  will  be  determined  by  the  data  that  is  fed  into 
the  model.  The  next  chapter  will  aid  this  process  by  determining  the  responsiveness  of 
these  data  parameters  and  locating  the  most  influential  data  parameters  on  the 
engagement  outcome. 

4.7  Summary 

The  STORM  air-to-air  adjudicator  has  improved  upon  the  deficiencies  of 
THUNDER.  The  particular  areas  of  improvement  are: 

•  The  adjudicator’s  assessment  of  the  volley  logic  overcomes  THUNDER’S 
problem  of  modeling  munitions  consumption  in  disparate  force  ratio 
scenarios. 
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•  The  concept  of  the  unanswered  volley  takes  into  account  the  opponent’s 
weapon  type  and  improves  THUNDER’S  probability  of  kill  calculation. 

•  It  allows  for  a  more  realistic  representation  of  the  Air  Force  doctrine  of  a 
SHOOT-LOOK-SHOOT  firing  strategy  instead  of  a  SHOOT-SHOOT 
strategy. 

Some  areas  of  possible  improvement  are: 

•  A  goal  of  the  adjudicator  is  to  have  data  that  is  tied  to  data  repositories  and 
have  links  to  real  world  systems.  The  source  of  many  of  these  data 
requirements  is  questionable  and  seems  subjective  in  nature.  Combination  of 
SME  and  calibration  methods  will  be  necessary  to  link  to  higher  fidelity 
models  such  as  BRAWLER.  The  data  input  to  the  STORM  adjudicator  is 
important  because  its  ability  to  match  reality  will  be  determined  by  the  data 
fed  into  the  model. 
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Table  4.  Comments  on  STORM  Data  Requirements  [13] 
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2.  Max  Number  tgts  SME  Unclass.  None  In  BRAWLER,  dependent  on  fire  control 

/volley  by  aircraft  by  Threat  -  Intel  equipment  and  pilot  skill  level.  Not  modeled  in 

munition  THUNDER. 
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by  phase  by  munition 
by  tactic  vs.  particular 
weapon 
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5.  Sensitivity  Analysis 


5.1  Introduction 

In  order  to  evaluate  the  effectiveness  of  the  implementation  of  STORM’s  air-to- 
air  adjudicator,  an  experiment  was  conducted  examining  the  sensitivity  of  various  user 
input  data.  The  purpose  of  the  experiment  is  twofold.  First,  determine  if  the  algorithm  is 
sensitive  to  air  platform  changes  so  that  it  can  be  validly  used  for  comparing  competing 
systems  and  strategies.  Second,  identify  the  most  influential  factors.  Due  to  the 
synergistic  effect  of  the  varying  input  values,  it  is  difficult  to  say  without  experimentation 
which  ones  are  most  influential.  Once  the  most  influential  factors  are  identified,  the 
developers  and  the  users  in  the  field  will  know  which  “control  knobs”  are  most  important 
in  determining  the  outcome  of  the  engagement(s). 

5.2  Scenarios 

Two  phases,  BVR  and  WVR,  and  two  tactic  types,  AVERSE  and  TOLERANT 
were  used  for  this  experiment.  In  order  to  evaluate  various  control  settings,  a  standard 
engagement  was  used  composed  of8F-15C’s  versus  8  MIG-29 ’s,  with  the  MIG-29 
having  tolerant  tactics.  The  number  of  aircraft  was  chosen  so  that  the  engagement  kills 
on  either  side  would  not  plateau  at  zero.  The  MIG-29 ’s  armament  is  composed  of  4  AA- 
10s,  4  AA-1  Is,  and  5  gun  salvos.  Each  F-15  is  carrying  4  AIM-120  AMRAAM  long- 
range  missiles,  4  AIM-9X  missiles,  and  5  gun  salvos.  The  MIG-29 ’s  control  settings 
were  left  at  reasonable  settings  and  were  not  changed  throughout  the  experiment.  It  was 
decided  to  change  the  input  settings  for  only  the  AIM- 120  AMRAAM  due  to  the 
combinatorial  nature  of  the  problem. 
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5.3  Design  of  Experiment 

The  design  of  the  experiment  was  a  2-level  half  fraction  design  with  seven  factors 
-  maximum  targets  per  volley  (Mtgts),  maximum  weapons  per  target  per  volley 
(Mwpns),  air-to-air  configuration  score  (Score),  probability  of  weapon  effectiveness 
(Pwe),  maximum  volleys(MaxVol),  probability  of  an  unanswered  volley  (Puv),  and 
tactic. 

When  one  can  assume  that  certain  high-order  interactions  are  negligible,  then 
information  on  the  main  effects  and  low-order  interactions  may  be  obtained  by  running  a 
fraction  of  the  complete  factorial.  The  one-half  fraction  of  the  27  design  was  chosen  and 
results  in  a  resolution  VII  experiment.  This  resolution  number  indicates  that  main  effects 
are  aliased  with  six  factor  interactions  and  higher.  Two  factor  interactions  are  aliased 
with  four  factor  interactions  and  higher.  Three  factor  interactions  are  aliased  with  other 
three  factor  interactions  and  higher.  Therefore,  a  resolution  VII  is  sufficient  to  determine 
the  main  effects  and  any  low  order  interactions.  A  27'1  experiment  results  in  64  total  runs 
[21:134,158]. 

The  design  matrix  is  shown  in  Table  5.  Minus  values  signify  low  settings  and 
positive  values  signify  high  settings.  Column  G,  tactic,  is  defined  by  the  relation,  G  = 
±ABCDEF. 
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Table  5.  Design  Matrix 


A 

B 

C 

D 

E 

F 

G 

Mtgt 

Mwpn 

Score 

Pwe  MaxVol 

Puv  Tactic 

1  -1 

-1 

-1 

-1 

-1 

-1 

1 

2  1 

-1 

-1 

-1 

-1 

-1 

-1 

3  -1 

1 

-1 

-1 

-1 

-1 

-1 

4  1 

1 

-1 

-1 

-1 

-1 

1 

5  -1 

-1 

1 

-1 

-1 

-1 

-1 

6  1 

-1 

1 

-1 

-1 

-1 

1 

62  1 

-1 

1 

1 

1 

1 

-1 

63  -1 

1 

1 

1 

1 

1 

-1 

64  1 

1 

1 

1 

1 

1 

1 

Mtgt  =  Max  number  of  targets  per 

Pwe  = 

=  Probability  of  weapons 

volley 

effectiveness 

Mwpn  =  Max  number  of  weapons  per 

MaxVol  =  Maximum  number  of 

target 

volleys 

Score  =  Air-to-air  configuration  score 

Puv  = 

Probability  of  an  unanswered 

volley 

Because  the  variability  of  the  sample  runs  was  unknown,  approximately  30  replications 


were  done  for  each  of  the  64  runs. 


5.4  Parameters 

Table  6  summarizes  the  factor  settings  used  corresponding  to  the  matrix  of  Table  5. 

Table  6.  Factor  Settings 


Factors 

Mtgt 

Mwpn 

Score 

Pwe 

MaxVol 

Puv 

Tactic 

High 

4 

2 

75 

0.75 

2.5 

0.7 

Tolerant 

Low 

3 

1 

55 

0.3 

1 

0.3 

Averse 
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The  “low”  setting  represents  this  study’s  worst  airframe/armament  capability, 
with  “high”  being  the  best.  The  settings  were  chosen  based  on  a  data  file  delivered  with 
the  adjudicator  prototype. 

5.5  Measure  of  Effectiveness 

One  measure  of  effectiveness  was  chosen  to  best  demonstrate  sensitivity  to 
various  input  changes.  The  outcome  that  is  of  most  interest  to  a  user  of  this  algorithm  is 
the  number  of  kills  on  each  side.  Although  blue  and  red  kills  are  of  interest  by 
themselves,  the  exchange  ratio  takes  both  of  them  into  account.  The  formula  for  the 
exchange  ratio  is: 

Exchange  Ratio  =  Blue  Kills  /  Red  Kills  (34) 

5.6  Least  Squares  Linear  Regression 

Linear  regression  involves  using  independent  predictor  variables  to  estimate  a 
model  in  order  to  predict  a  response.  A  first-order  model  is  represented  by  the  following 
equation: 

Y  =  p0  +  piXi  +  p2X2  +  ...  +  PkXk  +  e  (35> 

where  Y  is  the  response  variable,  x;’s  are  the  independent  variables,  and  Pi’s  are  the 
regression  coefficients  which  represent  the  expected  change  in  response  Y  per  unit 
change  in  Xj  when  all  remaining  independent  variables  Xj  (i^j)  are  held  constant,  s  is  the 
error  term  which  is  assumed  to  be  independent  and  identically  normally  distributed  with  a 
mean  of  zero  and  constant  variance  [19:1 10]. 
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5.6.1  Testing  for  Significance  of  Individual  Regression  Coefficients 


A  t-test  was  used  to  ensure  that  only  significant  variables  are  included  in  the 
model.  The  null  hypothesis  is  H0:  Pj  =  0,  and  the  alternative  hypothesis  is  HA:  pj  *  0.  If 
the  null  hypothesis  is  not  rejected,  the  coefficient,  pj,  for  the  Xj  variable  is  not  significant, 
and  Xj  is  removed  from  the  model.  The  test  statistic  is  given  by: 

b.  (36) 

+  —  _ l _ . 


where  bj  is  the  estimate  of  Pj,  and  Cjj  is  the  diagonal  element  of  (X’X)'1  corresponding  to 
bj,  where  X  represents  the  matrix  of  independent  x  variables.  The  null  hypothesis  is 
rejected  if  |to|  >  ta/2,n-k-i ,  where  n  is  sample  size  and  k  is  the  number  of  independent 
variables.  A  p-value  was  used  to  determine  whether  or  not  to  include  a  variable  in  the 
model.  A  p-value  is  the  smallest  value  of  a  for  which  the  null  hypothesis  can  be  rejected 
[19:111].  For  this  study,  a  variable  with  a  p-value  of  greater  than  0.10  was  considered 
insignificant. 

5.6.2  Testing  for  Significance  of  Regression 

It  is  also  necessary  to  test  whether  the  pj  coefficients  are  significant.  The  null 
hypothesis  is  H0:  Pi  =  P2  =  . . .  =  Pk  =  0,  with  an  alternative  of  Ha:  Not  all  pi  equal  zero. 
Rejecting  the  null  hypothesis  implies  that  at  least  one  of  the  independent  variables  is 
significant  to  the  model.  The  test  statistic  is: 
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F  = 


(37) 


SSR/k  _  MSr 
SSEl{n-k-\)~  MS E 


where  SSr  is  the  sum  of  squares  due  to  the  model  (or  regression)  and  SSe  is  the  sum  of 
squares  due  to  the  residual  (or  error).  The  formulas  for  these  estimates  are: 

(n  V  (38) 

I* 

SS„  =b'X'y~— — — 
n 

SSE  =  y'y-b'X'y  (39) 


H0  is  rejected  if  F0  >  Fa,  k,  n-k-i-  A  p-value  is  computed  for  this  test  statistic  to 
determine  whether  or  not  the  model  is  significant.  In  this  study,  the  model  was 
considered  significant  if  the  p-value  is  less  than  0.05. 

The  coefficient  of  multiple  determination,  R2,  is  another  estimate  that  implies  the 
appropriateness  of  the  model.  R2  is  the  measure  of  the  amount  of  reduction  in  the 
variability  of  the  response  obtained  from  the  independent  variables  used.  The  formula  for 
R2  is: 

Ri  SSj,  (40) 

SSR  +  SSE 

However,  adding  an  independent  variable  will  always  increase  R2,  so  a  better 
measure  is  given  by  an  adjusted  R2,  which  does  not  always  increase  with  the  addition  of  a 
variable.  The  formula  for  R2adj  is: 
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(41) 


R2  _1  SSs/(n-p) 

(SSR+SSE)/(n- 1) 

where p  is  the  number  of  Pi’s.  Higher  R2adj  values  indicate  a  better  fit  to  the  data. 

To  verify  that  model  assumptions  are  not  violated,  residual  analysis  was 
performed.  This  involves  verifying  that  the  error  terms,  efs,  are  independent  and 
identically  normally  distributed  with  mean  zero  and  constant  variance.  For  this  study,  a 
scatter  plot  of  the  residuals  against  the  predicted  values  was  used  to  visually  verify 
independence  and  constant  variance.  The  Shapiro-Wilk  test  was  used  to  verify  normality. 
The  null  hypothesis  of  the  test  is  that  the  distribution  is  normal.  For  this  study,  residuals 
with  a  p-value  of  less  than  0.05  were  considered  non-normal  [19:112]. 

5.7  Results 

There  was  some  question  as  whether  to  measure  the  number  of  kills  on  each  side 
for  just  the  phase  that  was  being  altered  or  whether  to  measure  the  number  of  kills  across 
all  phases.  In  order  to  capture  the  effects  of  one  phase’s  outcome  influencing  the  next, 
the  number  of  kills  and  hence  the  exchange  ratio  was  measured  across  all  phases. 

Based  on  several  trial  runs,  it  was  determined  that  the  overall  experiment  should 
be  split  into  three  separate  experiments.  One  experiment  would  measure  the  exchange 
ratio  while  changing  the  input  parameter  to  the  AMRAAM  in  the  BVR  phase.  Since 
there  are  no  other  missiles  fired  by  the  F-l  5  in  the  BVR  phase,  this  would  be  adequate. 
Another  experiment  would  measure  the  exchange  ratio  as  the  input  parameters  are 
changed  on  the  AMRAAM  fired  in  the  WVR  phase  first  on  the  priority  list.  Another 
experiment  would  measure  the  effects  of  the  AMRAAM  second  on  the  priority  list  and 
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the  AIM-9X  being  first  in  the  WVR  phase.  An  example  of  this  priority  list  is  shown  in 
Attachment  C.  As  discussed  previously,  the  munition  types  that  are  able  to  be  fired  from 
the  particular  platform  are  chosen  from  this  priority  list  until  that  particular  munition  type 
runs  out.  Then,  the  next  munition  type  is  chosen. 


5.7.1  Response  with  Changes  in  the  BVR  Phase 

A  transformation  of  the  responses  was  performed  to  obtain  a  predictive  model. 
Transformations  are  often  necessary  to  deal  with  problems  of  non-normality  of  residuals 
[22:296] .  The  transformation  used  was  the  inverse  transformation  of  the  responses. 
Table  7  summarizes  the  significant  factors  in  terms  of  the  coded  factors. 

In  terms  of  actual  factors,  two  equations  are  necessary  because  tactic  is  a 
significant  qualitative  variable  instead  of  a  quantitative  variable.  The  response  models 
are  as  follows. 

When  tactics  are  set  low  (averse),  the  equation  is: 
j  =  0.58  -  0.034 -Mtgt  -  0.068  •  Mwpn  -  0.17  •  Pwe  -  0.13  •  MaxVol  -  0.084  •  Puv 


Table  7.  Significant  Factors  (BVR) 


Factor 

Coefficient 

Estimate 

DF 

Standard  t  for  HO 
Error  Coeff=0 

Prob  >  |t| 

Intercept 

0.640336 

1 

0.012468 

Mtgt 

-0.03442 

1 

0.012468  -2.76108 

0.0077 

Mwpn 

-0.06845 

1 

0.012468  -5.49012 

<  0.0001 

Pwe 

-0.16971 

1 

0.012468  -13.6118 

<  0.0001 

MaxVol 

-0.13347 

1 

0.012468  -10.7052 

<  0.0001 

Puv 

-0.08397 

1 

0.012468  -6.73495 

<  0.0001 

Tactic 

0.056103 

1 

0.012468  4.4999 

<  0.0001 
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When  tactics  are  high  (tolerant),  the  equation  is: 


j  =  0.70  -  0.034  •  Mtgt  -  0.068  •  Mwpn  -  0. 1 7  •  Pwe  -  0. 1 3  •  MaxVol  -  0.084  •  Puv 

The  R2adj  (Table  8)  is  relatively  high,  and  the  model  is  significant  as  seen  in  Table  9. 

The  most  influential  factors,  when  the  AIM- 120  is  varied  using  the  high  and  low 
settings,  are  (in  order  of  influence): 

1 .  Probability  of  weapons  effectiveness 

2.  Maximum  volleys 

3 .  Probability  of  an  unanswered  volley 

4.  Maximum  weapons  per  target  per  volley 

5.  Maximum  targets  per  volley 


Table  8.  BVR  Summary  of  Fit 


R-Squared 

0.876156 

Adj  R-Squared 

0.86312 

Root  MSE 

0.099741 

Table  9.  BVR  Analysis  of  Variance 


Source 

Sum  of 

Mean 

F 

Model 

Squares 

4.01172 

DF 

6 

Square 

0.668619 

Value 

67.2096 

Prob  >  F 

<  0.0001 

Residual 

0.567052 

57 

0.009948 

Cor  Total 

4.57877 

63 

(43) 
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5.7.2  Response  with  Changes  in  the  WVR  Phase,  First  on  Priority  List 

The  experiment  was  run  inside  the  WVR  phase  while  changing  the  AMRAAM 
first  on  the  priority  list.  This  priority  list  is  shown  in  Attachment  C.  Once  it  was 
determined  from  the  model  above  that  two  factor  interactions  were  not  significant,  the 
experiment  was  changed  to  a  27'3  resulting  in  a  resolution  IV  experiment,  making  data 
gathering  easier.  This  resulted  in  the  design  matrix  in  Table  10. 


Table  10.  Amended  Design  Matrix 


A 

Mtgt 

B 

Mwpn 

C 

Score 

D 

Pwe 

E 

MaxVol 

F 

Puv 

G 

Tactic 

1 

-1 

-1 

-1 

-1 

-1 

-1 

-1 

2 

1 

-1 

-1 

-1 

1 

-1 

1 

3 

-1 

1 

-1 

-1 

1 

1 

-1 

4 

1 

1 

-1 

-1 

-1 

1 

1 

5 

-1 

-1 

1 

-1 

1 

1 

1 

6 

1 

-1 

1 

-1 

-1 

1 

-1 

7 

-1 

1 

1 

-1 

-1 

-1 

1 

8 

1 

1 

1 

-1 

1 

-1 

-1 

9 

-1 

-1 

-1 

1 

-1 

1 

1 

10 

1 

-1 

-1 

1 

1 

1 

-1 

11 

-1 

1 

-1 

1 

1 

-1 

1 

12 

1 

1 

-1 

1 

-1 

-1 

-1 

13 

-1 

-1 

1 

1 

1 

-1 

-1 

14 

1 

-1 

1 

1 

-1 

-1 

1 

15 

-1 

1 

1 

1 

-1 

1 

-1 

16 

1 

1 

1 

1 

1 

1 

1 
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An  inverse  transformation  was  also  used  for  this  portion  of  the  experiment.  Based 


on  this  transformation,  Table  1 1  shows  the  significant  factors. 


The  response  models  in  terms  of  actual  factors  are  as  follows: 


When  tactics  are  set  low  (averse),  the  equation  is: 


1  (44) 

y  =  0.70-  0.66  -MaxWpn 


When  tactics  are  high  (tolerant),  the  equation  is: 


1  (45) 

y  =  0.40  -0.66-  MaxWpn 


The  R  adj  (Table  12)  is  high,  and  the  model  is  significant  as  seen  in  Table  13. 


Table  11.  Significant  Factors  (WVR  -  First  on  Priority  List) 


Factor 

Coefficient 

Estimate 

DF 

Standard  t  for  HO 
Error  Coeff=0 

Prob  >  |t| 

Intercept 

0.552916 

1 

0.013983 

Mwpn 

-0.06583 

1 

0.013983  -4.70765 

0.0004 

Tactic 

-0.1518 

1 

0.013983  -10.8561 

<  0.0001 

Table  12.  WVR  (First  on  Priority  List)  Summary  of  Fit 


R-Squared 

0.915042 

Adj  R-Squared 

0.901972 

Root  MSE 

0.055932 
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Table  13.  WVR  (First  on  Priority  List)  Analysis  of  Variance 


Source 

Sum  of 
Squares 

DF 

Mean 

Square 

F 

Value 

Prob  >  F 

Model 

0.438026 

2 

0.219013 

70.0088 

<  0.0001 

Residual 

0.040669 

13 

0.003128 

Cor  Total 

0.478695 

15 

When  the  AIM- 120’s  settings  are  varied  in  the  WVR  and  it  is  first  on  the  priority 
list,  maximum  weapons  per  target  per  volley  and  tactic  are  the  most  influential.  Since  a 
majority  of  the  AIM-120s  are  shot  in  the  BVR  stage,  it  makes  sense  that  a  majority  of  the 
influential  factors  in  the  BVR  experiment  will  drop  out.  Also,  since  changing  the  tactic 
in  file  WVR  phase  means  changing  the  tactic  in  all  phases  (a  plane  has  the  same  tactic  for 
all  phases),  tactic  should  be  a  significant  factor  no  matter  what. 

5.7.3  Response  with  Changes  in  the  WVR  Phase,  Second  on  Priority  List 

Again,  an  inverse  transformation  was  used  for  this  portion  of  the  experiment. 
Table  14  shows  the  significant  factor  based  on  this  transformation: 


Table  14.  Significant  Factors  (WVR  -  Second  on  Priority  List) 


Coefficient 

Standard 

tfor  HO 

Factor 

Estimate 

DF 

Error 

Coeff=0 

Prob  >  |t| 

Intercept 

G-Tactic 

0.5711 

-0.15854 

1 

1 

0.014419 

0.014419 

-10.9959 

<  0.0001 
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The  response  models  in  terms  of  actual  factors  are  as  follows. 


When  tactics  are  set  low  (averse),  the  equation  is: 


-  =  0.70 
Y 


When  tactics  are  high  (tolerant),  the  equation  is: 


-  =  0.40 
Y 


(46) 


(47) 


The  R2adj  (Table  15)  is  relatively  high,  and  the  model  is  significant  as  seen  in  Table  16. 

As  hypothesized  in  the  previous  experiment,  when  the  AIM-120’s  settings  are 
varied  in  the  WVR  and  it  is  second  on  the  priority  list,  tactic  is  now  the  most  influential. 
By  the  time  that  platform  gets  to  shooting  the  AIM- 120,  it  has  shot  most  of  the  other 


Table  15.  WVR  (Second  on  Priority  List)  Summary  of  Fit 


R-Squared 

0.896227 

Adj  R-Squared 

0.888815 

Root  MSE 

0.057674 

Table  16.  WVR  (Second  on  Priority  List)  Analysis  of  Variance 


Source 

Sum  of 
Squares 

DF 

Mean 

Square 

F 

Value 

Prob  >  F 

Model 

0.402181 

1 

0.402181 

120.91 

<  0.0001 

Residual 

0.046568 

14 

0.003326 

Cor  Total 

0.448749 

15 
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missiles  and  achieved  most  of  the  kills.  As  stated  previously,  it  makes  sense  that  tactic  is 
most  important  because  an  aircraft’s  tactic  affects  all  of  the  phase  outcomes. 

5.8  Summary 

Based  on  the  results  shown  in  this  chapter,  the  battle  outcomes  are  responsive  to 
changes  in  the  air-to-air  parameters.  The  probability  of  weapons  effectiveness,  maximum 
volleys,  and  probability  of  an  unanswered  volley  are  the  top  three  most  influential  on  the 
outcome.  This  demonstrates  the  algorithm’s  ability  to  be  used  for  comparisons  and 
suggests  which  “control  knobs”  are  most  influential  for  the  user. 
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6.  Conclusions  and  Recommendations 


6.1  Validation  Assessment  of  the  STORM  Air-to-Air  Adjudicator 

This  thesis  effort  has  approached  evaluation  of  this  model  from  two  angles.  First, 
the  algorithm  itself  underwent  structural  validation  to  identify  any  limitations  and 
restrictive  assumptions.  It  was  compared  to  the  real  world  keeping  in  mind  the  intended 
use  of  the  model  for  campaign  level  analysis  only.  It  was  also  compared  to  its 
predecessor,  THUNDER,  to  ensure  that  it  improved  upon  its  weaknesses.  Second,  the 
result  validation  effort  culminated  in  a  design  of  experiment  to  evaluate  the 
responsiveness  of  the  model. 

Data  validation  was  accomplished  to  see  if  the  data  was  available  and  appropriate. 
It  was  discovered  that  the  adjudicator  is  largely  data  driven,  and  particular  care  should  be 
taken  to  ensure  that  the  data  is  accurate.  It  seems  much  of  the  data  required  will  be 
available  from  higher  fidelity  models  while  other  data  requirements  seem  very  subjective 
in  nature. 

Sensitivity  analysis  was  used  to  determine  the  most  influential  input  variables  on 
the  outcome  of  the  engagement.  It  was  determined  that  the  probability  of  weapons 
effectiveness,  maximum  volleys,  and  probability  of  an  unanswered  volley  are  the  top 
three  most  influential  on  the  outcome.  This  demonstrates  the  algorithm’s  ability  to  be 
used  for  comparisons  and  suggests  which  “control  knobs”  are  most  influential  for  the 
user. 

The  bottom  line  is  that  STORM  air-to-air  adjudicator  will  prove  to  be  a  useful 
tool  for  campaign  analysis  in  the  new  century.  The  review  of  the  components  of  the 
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algorithm  indicates  that  it  has  implemented  the  air-to-air  combat  process  with  a 
considerable  degree  of  fidelity,  although  improvements  can  still  be  made.  Although  there 
are  assumptions  related  to  aggregation,  it  still  reasonably  represents  the  elements  of  the 
air-to-air  combat  environment.  Some  of  the  aspects  of  air-to-air  combat  STORM  models 
particularly  well  are: 

•  The  phase  concept  improves  overall  transparency  and  captures  most  of  the 
issues  in  the  air-to-air  combat  environment  appropriate  to  a  campaign  level 
model. 

•  The  volley  concept  accurately  represents  munitions  consumption. 

•  The  probability  of  an  unanswered  volley  realistically  accounts  for  range 
advantages  and  can  be  tied  to  aircraft  platform. 

•  The  parameters  for  the  mean  and  standard  deviation  of  the  perceived  number 
of  aircraft  is  accurately  computed. 

•  The  ability  to  escape  is  based  on  an  innovative  function  that  can  be  shaped  by 
the  user-input  values. 

•  Pwe  can  be  tied  to  the  shooting  aircraft  platform. 

•  The  tracking  of  lost  munitions  of  killed  aircraft  is  accurately  accounted  for. 

•  The  adjudicator’s  assessment  of  the  volley  logic  overcomes  THUNDER’S 
problem  of  modeling  munitions  consumption  in  disparate  force  ratio 
scenarios. 
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•  The  concept  of  the  unanswered  volley  takes  into  account  the  opponent’s 
weapon  type  and  improves  THUNDER’S  probability  of  kill  calculation. 

•  STORM  allows  for  a  more  realistic  representation  of  the  Air  Force  doctrine  of 
a  SHOOT-LOOK-SHOOT  firing  strategy  instead  of  a  SHOOT-SHOOT 
strategy. 

6.2  Recommendations 

It  cannot  be  over  emphasized  that  the  most  important  recommendation  for 
improvement  of  the  STORM  air-to-air  adjudicator  is  to  place  particular  emphasis  on  the 
development  of  the  data  input.  The  acceptability  of  the  output  of  engagements  is 
dependent  on  the  accuracy  of  the  input  data. 

Other  particular  areas  of  improvement  are: 

•  Definition  of  phases  should  be  determined  and  agreed  upon  by  the  users  of 
STORM. 

•  Certain  issues  of  time  and  distance  should  be  modeled  either  inside  or  outside 
the  adjudicator  (e.g.  fuel  constraints  and  position  relative  to  the  FLOT  affect 
engagement  strategies) 

•  The  perceived  score  of  an  opponent  is  based  on  the  actual  total  number  of 
aircraft  instead  of  the  perceived  number  of  aircraft.  Should  be  based  on  what 
the  friendly  believes  the  opponent  has. 
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•  There  is  some  concern  that  Pwe  would  promote  “double  dipping”  with  the 
probability  of  an  unanswered  volley  unless  the  definitions  are  clearly 
understood  by  users  of  the  adjudicator. 

•  The  targeting  decision  does  not  include  circumstances  involving  HVAs.  An 
HVA  may  have  a  low  aircraft  configuration  score  and  would  therefore  be 
lower  in  priority  to  target  by  the  opponent. 

•  A  goal  of  the  adjudicator  was  have  data  tied  to  data  repositories  with  links  to 
real  world  systems.  The  source  of  many  of  these  data  requirements  is 
questionable  and  seems  subjective  in  nature.  Combination  of  SME  and 
calibration  methods  are  necessary  to  link  to  higher  fidelity  models  such  as 
BRAWLER.  The  data  input  to  the  STORM  adjudicator  is  important  because 
its  ability  to  match  reality  will  be  determined  by  the  data  fed  into  the  model. 

6.3  Final  Thought 

The  STORM  air-to-air  adjudicator  is  an  innovative  model  to  meet  the  needs  of 
campaign  level  air-to-air  analysis.  Any  deficiencies  listed  in  this  thesis  are  opportunities 
for  improvement  when  the  first  version  of  STORM  is  released. 
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Appendix  A:  THUNDER  Air-to-Air  Missions 


In  accordance  with  USAF  doctrine,  THUNDER  missions  are  defined  by  their 
objectives  rather  than  the  type  of  aircraft  accomplishing  the  mission.  THUNDER  defines 
27  missions,  grouped  as  air-to-ground  missions,  air-to-air  missions,  suppression  of  enemy 
air  defense  (SEAD)  missions,  and  High  Value  Asset  (HVA)  missions.  The  air-to-air 
missions  defined  in  THUNDER  are  described  in  the  following  sections. 

A.l  Barrier  Combat  Air  Patrol  (BARCAP) 

A  BARCAP  aircraft,  while  on-station,  patrols  a  designated  area  on  its  own  side  of 
the  forward  line  of  troops  (FLOT)  in  order  to  intercept  any  enemy  aircraft  that  attempt  to 
pass  through  the  area.  It  does  not  try  to  intercept  specific  threats  but,  instead,  patrols  a 
rectangular  area,  called  the  Combat  Area  Patrol  (CAP),  on  its  own  side  of  the  FLOT. 
Upon  passing  the  BARCAP  area,  a  threat  is  subject  to  detection  by  the  BARCAP  aircraft. 
If  the  threat  is  detected,  an  engagement  will  occur.  Multiple  enemy  flight  groups  in  the 
BARCAP  area  at  the  same  time  are  treated  independently.  No  saturation  effect  is 
considered.  After  the  engagement,  surviving  aircraft  with  air-to-air  munitions  remaining, 
continue  to  form  a  CAP.  When  it  can  no  long  be  maintained  due  to  lack  of  fuel/weapons, 
the  aircraft  return  to  base  [26:25,  35-37]. 

A.2  Defensive  Counter-Air  (DCA) 

A  DCA  aircraft  sits  on  strip  alert  and  flies  in  response  to  the  enemy’s  penetrating 
attack.  The  DCA  is  triggered  by  detection  of  the  ingressing  enemy  flight  group.  If  the 
aircraft  type’s  probability  of  enemy  interceptor  launch  of  the  primary  flight  is  less  than 
one,  a  random  draw  is  made  to  determine  if  the  ingressing  flight  group  will  even  be 
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considered  for  DCA.  If  selected,  a  search  is  made  of  all  possible  interceptors  for  the 
earliest  possible  intercept.  If  the  side  intercept  ratio  is  not  satisfied,  more  aircraft  are 
sent,  from  possibly  different  bases.  The  DCA  group  will  then  fly  to  the  predicted 
intercept  point.  It  then  checks  to  see  if  the  enemy  flight  group  is  within  radar  detection. 

If  detected,  THUNDER  decides  if  engagement  occurs  using  the  air-to-air  engagement 
probability.  If  it  takes  off  and  fails  to  intercept  the  enemy,  it  acts  exactly  like  a  BARCAP 
aircraft,  using  its  intercept  point  as  the  center  of  an  orbit  and  circling  on-station,  waiting 
for  threats,  as  long  as  fuel  and  weapons  permit.  When  the  mission  is  completed  the  flights 
return  to  their  bases  [26:25,34]. 

A.3  Over-The-Flot  Defensive  Counter-Air  (ODCA) 

An  ODCA  aircraft  sits  on  strip  alert  and  waits  for  a  certain  type  of  mission  and 
size  of  enemy  flight  to  takeoff.  It  engages  enemy  aircraft  before  it  enters  friendly  air 
space  by  penetrating  a  user-defined  distance  into  enemy  territory  to  attack  the  aircraft.  If 
the  ODCA  aircraft  takes  off  and  fails  to  intercept,  it  is  able  to  begin  a  user-defined  orbit 
over  enemy  territory,  circling  on  station  and  waiting  for  threats.  The  user  can  determine 
that  the  aircraft  will  not  orbit  and  should  return  to  base  [26:26]. 

ODCA  is  similar  to  DCA  missions  in  that  they  are  both  reactive  missions.  The 
only  difference  is  the  ODCA  attempts  to  engage  enemy  flights  before  they  enter  friendly 
airspace.  This  concept  was  designed  for  very  fast,  long  range,  survivable  interceptors. 

A.4  High  Valued  Asset  Attack  (HVAA) 

An  HVAA  aircraft  may  be  triggered  when  it  first  goes  on  alert,  when  an  enemy 
HVA  aircraft  first  arrives  at  its  orbit,  or  when  an  enemy  HVA  aircraft  survives  another 
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HVAA  attack.  HVA  flights  are  planned  to  cover  a  predefined  set  of  orbit  points.  Flights 
are  assigned  to  orbit  points  using  a  priority  system.  Air  combat  avoidance  behavior  is 
triggered  using  a  user-defined  “safety  zone.”  User  inputs  defined  which  enemy  missions 
are  deemed  threatening.  When  an  HVA  aircraft’s  safety  zone  is  penetrated,  the  HVA 
flight  goes  off-station  for  a  duration  determined  by  a  user-input  random  function.  If  an 
HVA  flight  is  destroyed,  its  orbit  point  is  covered  by  an  alert  HVA  if  one  exists.  HVA 
flights  return  home  after  their  orbit  time  has  elapsed. 

There  are  three  events  that  may  trigger  an  HVAA  flight  against  an  HVA  type 
flight.  The  first  event  occurs  when  a  HVAA  flight  comes  on  alert.  The  second  event 
occurs  when  an  HVA  type  flight  begins  an  orbit.  The  final  event  occurs  when  an  HVAA 
flight  fails  to  intercept  a  HVA  flight.  When  a  target  HVA  flight  has  been  identified  as  a 
target,  an  HVAA  flight  on  alert  will  build  a  flight  path  to  the  HVA  flight’s  orbit  point. 
The  HVAA  flight  intercept  point  is  the  HVA  orbit  point.  If  an  HVA  mission  is  at  its 
orbit  point  and  is  avoiding  air  combat  when  it  is  intercepted  by  HVAA,  then  a  special 
attrition  methodology  is  applied.  A  random  draw  against  the  interceptor’s  probability  of 
detection  is  used  to  determine  if  a  successful  intercept  occurred.  If  so  the  number  of 
shots  per  engagement  and  air-to-air  Pk’s  are  applied  directly,  bypassing  the  normal  air-to- 
air  engagement  logic.  In  all  other  air-to-air  situations  involving  HVAs,  the  normal 
engagement  logic  is  applied  [26:26,40]. 

A.5  Air-to-air  Escort  (AIRESC) 

A  close  escort  mission  accompanies  a  strike  package  over  enemy  territory. 

During  an  engagement,  escorts  may  be  attacked  by  enemy  fighters.  Escorts  may  either 
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split  off  to  engage  enemy  fighters  or  stay  with  the  strike  aircraft.  Air  escort  aircraft  are 
assigned  to  flight  groups  based  on  the  priority  for  air  escort  support  of  the  flight  group’s 
primary  aircraft  type.  The  priority  for  escort  support  is  based  upon  several  factors  such 
as  the  type  of  aircraft  priority,  mission  importance,  and  depth  of  penetration  into  enemy 
territory  for  penetrating  missions  or  proximity  to  the  FLOT.  The  number  of  air  escort 
aircraft  to  accompany  a  flight  group  is  determined  by  user  input.  Air  escort  flights  will 
become  attached  to  the  flight  group  they  are  supporting.  They  will  engage  any  enemy 
interceptor  aircraft  that  pose  a  threat  to  the  flight  group.  Air  escort  aircraft  may  suffer 
area  air  defense  attrition  but  do  not  accompany  the  flight  group  into  the  terminal  phase  of 
a  ground  attack  [26:26,41]. 

A.6  Fighter  Sweep  (FSWP) 

A  detached  escort  fighter  mission  that  provides  air-to-air  protection  for  a  strike 
package  by  preceding  it  and  establishing  an  orbit  over  the  target.  FSWP  missions  are 
planned  to  be  executed  as  a  precursor  to  other  air-to-ground  missions  but  are  flown 
regardless  whether  the  ground  attack  missions  were  actually  launched.  They  may  be 
attacked  by  enemy  ODCA,  BARCAP,  and  DCA  flights.  After  a  FSWP  aircraft  proceeds 
to  its  target  at  a  user-defined  altitude,  it  begins  a  CAP  based  on  the  aircraft  type’s 
BARCAP  parameters.  It  orbits  for  a  specified  amount  of  time  and  uses  the  same 
engagement  logic  as  BARCAP.  It  attempts  to  engage  any  enemy  aircraft  that  pass 
through  the  patrol  area.  In  addition,  it  is  subject  to  interception  by  enemy  fighters  and 
engagement  by  enemy  surface-to-air  defenses  [26:39]. 
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A.7  Summary 

The  primary  goal  of  a  BARCAP,  DCA,  ODCA,  or  HVAA  mission  is  to  protect  a 
friendly  target.  Although  destruction  of  opponent  strike  or  HVA  aircraft  is  ideal, 
destruction  is  not  the  only  way  to  accomplish  the  mission  goal.  Merely  threatening  the 
opponent  may  cause  him  to  mission  abort.  The  friendly  aircraft  have  then  achieved  a 
‘mission  kill’  [8:105, 20:331]. 
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Appendix  B:  Example  of  Data  Input  File  “Type” 


typeairmunt :  12 
AMRAAM 
AIM54 
AIM7M 
AIM9X 
AIM9M 
GUN 
AA9 
AA12 
AA10 
AA11 
APEX 
ATOLL 

typeaa:  14 

F15C  elt_size :  X  %c2 :  XX 

#config:  X 
AA1  bingo :  XX 
X  AMRAAM 
X  AIM9X 
X  GUN 

F15E  elt_size :  X  %c2 :  XX 

#config:  X 
AA1  bingo:  XX 
X  AMRAAM 
X  AIM9X 
X  GUN 

AG1  bingo :  -X 
X  AMRAAM 
X  AIM9X 
X  GUN 

F15A  elt_size :  X  %c2 :  XX 

#config:  X 
AA1  bingo:  XX 
X  AIM7M 
X  AIM9M 
X  GUN 


* 

end 
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Appendix  C:  Example  of  Data  Input  File  “A2A” 


phases 

3 

VLR 

BVR 

WVR 

a2a_tactics 

3 

AVERSE 

MEDIUM 

TOLERANT 

a2  a_munt_pr  e  f  er  ence__by_j?has  e 
VLR 

AA9 

AIM54 

BVR 

AMRAAM 

AA12 

AIM7M 

AA10 

AA9 

AIM54 

APEX 

WVR 

AA11 

AIM9X 

AIM9M 

AMRAAM 

AIM7M 

AA12 

AA10 

APEX 

ATOLL 

GUN 

max_t  g  t  s  /  vo  1 1  ey_by__  typ  eaa_by__typea  i  rmun  t 


AMRAAM" 

AIM54 

AIM7M 

AIM9X 

AIM9M 

GUN 

AA9 

AA12 

AA10 

AA11 

F15C 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

F15E 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

F15A 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

F14C 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

F16B50 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

B1 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

SU2  7 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

SU24 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

MiG31 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

MiG2  9 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

MiG23 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

MiG21 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 
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max jwpns  /  tgt  /  vol  ley_by_typeaa_by_typeairmunt 


AMRAAM 

AIMS  4 

AIM7M 

AIM9X 

AIM9M 

GUN 

AA9 

AA12 

AA10 

AA11 

FISC 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

F15E 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

FI  5  A 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

F14C 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

F16B50 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

B1 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

SU2  7 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

SU24 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

MiG31 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

MiG2  9 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

MiG23 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

MiG21 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

a2a_score_by_typeaaJby_typeairmunt 


AMRAAM  AIMS  4  AIM7M  AIM9X 


F15C 

X 

X 

X 

X 

F15E 

X 

X 

X 

X 

FISA 

X 

X 

X 

X 

F14C 

X 

X 

X 

X 

F16B50 

X 

X 

X 

X 

B1 

X 

X 

X 

X 

SU27 

X 

X 

X 

X 

SU24 

X 

X 

X 

X 

MiG31 

X 

X 

X 

X 

MiG29 

X 

X 

X 

X 

MiG23 

X 

X 

X 

X 

MiG21 

X 

X 

X 

X 

AIM9M 

GUN 

AA9 

AA12 

AA10 

AA11 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

min_j?erceived_a2a_score_by_typeaa 


F15C  XX 
F15E  XX 
F15A  XX 
F14C  XX 
F16B50  XX 
B1  XX 
SU27  XX 
SU24  XX 
MiG31  XX 
MiG29  XX 
MiG23  XX 
MiG21  XX 


a2a_targets 

2 

FIGHTER 

FISC  FISA  F14C  SU27  MiG31  MiG29  MiG23  MiG21  end_typeaa 

BOMBER 

F15E  F16B50  B1  SU24  end_typeaa 
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a2a_shooters 

13 

15AMRAAM 


F15C  F15E 

end_typeaa 

AMRAAM 

end  typeairmunt 

16AMRAAM 

F16B50 

end_typeaa 

AMRAAM 

end_typeairmunt 

AIMS  4 

F14C 

end_typeaa 

AIM54 

end_typeai rmunt 

AIM7M 

F15A 

end_typeaa 

AIM7M 

end_typeai rmunt 

AIM9X 

F15C  F14C 

F15E  F16B50  end_typeaa 

AIM9X 

end_typeairmunt 

AIM9M 

F15A  F16B50  end_typeaa 

AIM9M 

end_typeai rmunt 

GUN 

F15C  F14C 

F15E  F15A  F16B50  REDMAX  SU27  SU24  MiG31  MiG29  MiG23 

MiG21 

end_typeaa 

AA9 

MiG3 1 

end_typeaa 

AA9 

end_typeai rmunt 

AA12 

SU2  7 

end_typeaa 

AA12 

end_typeairmunt 

AA10 

MiG2  9 

end_typeaa 

AA10 

end_typeai rmunt 

AA11 

SU27  SU24 

MiG31  MiG29  MiG23  end_typeaa 

AA11 

end_typeai rmunt 

APEX 

MiG23 

end_typeaa 

APEX 

end_typeai rmunt 

ATOLL 

MiG21 

end_typeaa 

ATOLL 

end_typeai rmunt 

pwe  by_phase, a2a_shooter , a2a_target , 

shooter  tactics ,  target__tactics 

VLR 

AIMS 4  FIGHTER 

{  AVERSE 

AVERSE 

XX 

AVERSE 

MEDIUM 

XX 

AVERSE 

TOLERANT 

XX 

MEDIUM 

AVERSE 

XX 

MEDIUM 

MEDIUM 

XX 
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MEDIUM 

TOLERANT 

XX 

TOLERANT 

AVERSE 

XX 

TOLERANT 

MEDIUM 

XX 

TOLERANT 

TOLERANT 

XX 

AIM54 

BOMBER 

{ 

AVERSE 

ALL 

XX 

MEDIUM 

ALL 

XX 

TOLERANT  ALL 

XX 

AIM54 

MAX 

{ 

AVERSE 

ALL 

XX 

MEDIUM 

ALL 

XX 

TOLERANT 

ALL 

XX 

AA9 

FIGHTER 

{ 

AVERSE 

AVERSE 

XX 

AVERSE 

MEDIUM 

XX 

AVERSE 

TOLERANT 

XX 

MEDIUM 

AVERSE 

XX 

MEDIUM 

MEDIUM 

XX 

MEDIUM 

TOLERANT 

XX 

TOLERANT 

AVERSE 

XX 

TOLERANT 

MEDIUM 

XX 

TOLERANT 

TOLERANT 

XX 

AA9 

BOMBER 

{ 

AVERSE 

ALL 

XX 

MEDIUM 

ALL 

XX 

TOLERANT 

ALL 

XX 

AA9 

MAX 

{ 

AVERSE 

ALL 

XX 

MEDIUM 

ALL 

XX 

TOLERANT 

ALL 

XX 

end_j?hase 

BVR 

AIM54 

FIGHTER 

{ 

AVERSE 

AVERSE 

XX 

AVERSE 

MEDIUM 

XX 

AVERSE 

TOLERANT 

XX 

MEDIUM 

AVERSE 

XX 

MEDIUM 

MEDIUM 

XX 

MEDIUM 

TOLERANT 

XX 

TOLERANT 

AVERSE 

XX 

TOLERANT 

MEDIUM 

XX 

TOLERANT 

TOLERANT 

XX 

AIM54 

BOMBER 

{ 

AVERSE 

ALL 

XX 

MEDIUM 

ALL 

XX 

TOLERANT 

ALL 

XX 

AIMS  4 

MAX 

{ 

AVERSE 

ALL 

XX 

MEDIUM 

ALL 

XX 

TOLERANT 

ALL 

XX 

15AMRAAM  FIGHTER 

{  AVERSE  AVERSE  XX 

AVERSE  MEDIUM  XX 
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AVERSE 

TOLERANT 

MEDIUM 

AVERSE 

MEDIUM 

MEDIUM 

MEDIUM 

TOLERANT 

TOLERANT 

AVERSE 

TOLERANT 

MEDIUM 

TOLERANT 

TOLERANT 

15AMRAAM  BOMBER 

{  AVERSE 

ALL 

MEDIUM 

ALL 

TOLERANT 

ALL 

16AMRAAM  FIGHTER 

{  AVERSE 

AVERSE 

AVERSE 

MEDIUM 

AVERSE 

TOLERANT 

MEDIUM 

AVERSE 

MEDIUM 

MEDIUM 

MEDIUM 

TOLERANT 

TOLERANT 

AVERSE 

TOLERANT 

MEDIUM 

TOLERANT 

TOLERANT 

16AMRAAM  BOMBER 

{  AVERSE 

ALL 

MEDIUM 

ALL 

TOLERANT 

ALL 

* 

* 

* 

end_phase 


WVR 

15AMRAAM  FIGHTER 


{  AVERSE 

AVERSE 

AVERSE 

MEDIUM 

AVERSE 

TOLERANT 

MEDIUM 

AVERSE 

MEDIUM 

MEDIUM 

MEDIUM 

TOLERANT 

TOLERANT 

AVERSE 

TOLERANT 

MEDIUM 

TOLERANT 

TOLERANT 

15AMRAAM  BOMBER 

{  AVERSE 

ALL 

MEDIUM 

ALL 

TOLERANT 

ALL 

15AMRAAM  MAX 

{  AVERSE 

ALL 

MEDIUM 

ALL 

TOLERANT 

ALL 

16AMRAAM  FIGHTER 

{  AVERSE 

AVERSE 

AVERSE 

MEDIUM 

AVERSE 

TOLERANT 

888  8  8  8  888  888888888  888  888888888  8  8  8  8  8  8  8  8  8  8 


* 


AIM9X 

FIGHTER 

{ 

AIM9X 

ALL 

BOMBER 

ALL 

XX 

{ 

AIM9M 

ALL 

FIGHTER 

ALL 

XX 

{ 

AIM9M 

ALL 

BOMBER 

ALL 

XX 

{ 

GUN 

ALL 

FIGHTER 

ALL 

XX 

{ 

GUN 

ALL 

BOMBER 

ALL 

XX 

{ 

* 

* 

* 

ALL 

ALL 

XX 

End_j?hase 

max_volleys (min, mode, max) _by_phase, a2a_shooter , a2a_tgt , shooter_tactics 
VLR 


AIM54 

{ 


AIM54 


{ 


FIGHTER 


AVERSE 

XX 

XX 

XX 

MEDIUM 

XX 

XX 

XX 

TOLERANT 

XX 

XX 

XX 

BOMBER 

ALL 

XX 

XX 

XX 

* 


* 


end_phase 


BVR 

* 

* 

* 

prob_unanswered_volley_by_phase, a2a_shooter , a2a_shooter , shooter_tactics 
VLR 


AIM54 

{ 

ALL 

AA9 

XX 

} 

AIM54 

{ 

ALL 

AIM54 

XX 

} 

AA9 

{ 

ALL 

AIMS  4 

XX 

} 

AA9 

{ 

ALL 

AA9 

XX 

} 

end_jphase 
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15AMRAAM  AA9 
{  AVERSE 

MEDIUM 
TOLERANT 
15AMRAAM  AA12 
{  AVERSE 

MEDIUM 
TOLERANT 
15AMRAAM  AA10 
{  AVERSE 

MEDIUM 
TOLERANT 
15AMRAAM  APEX 
{  AVERSE 

MEDIUM 
TOLERANT 


V*  K>*  M  K>  M 

n  kS  n  r'Si'Sr'S  i^SkSkS  kSkSkS 

U  V  U  kvt  U  U  tyl  lv<  Kj  lul 

n  n  n  KS  kS  kS  kS  k*S  »*S  kS  kS  rS 


Appendix  D:  Example  of  Data  Input  File  “Escape” 


a2a_escape_classes 

3 

4  GEN 

F15C  F15A  F14C  SU27  MiG31  MiG29  F15E  F16B50  end_typeaa 

3  GEN 

MiG23  MiG21  end_typeaa 

BOMBER 

B1  SU24  end_typeaa 


escapeDat (lowFR, Pesc, hiFR, hiPesc,bias) by_jphase, a2a_escape_class , a2a_esc 
ape_class 

©bias  of  -1.0  means  all  escape  opportunities  come  before  first  volley 
©bias  of  0.0  means  escape  opportunities  equally  distributed  amongst 
volleys 

©bias  of  1.0  means  all  escape  opportunities  come  after  last  volley 
VLR 


4  GEN 


4  GEN 

XX 

XX 

XX  XX 

XX 

3  GEN 

XX 

XX 

XX  XX 

XX 

-X 

BOMBER 

XX 

XX 

XX  XX 

XX 

-X 

3  GEN 

4  GEN 

XX 

XX 

XX  XX 

XX 

-X 

3  GEN 

XX 

XX 

XX  XX 

XX 

-X 

BOMBER 

XX 

XX 

XX  XX 

XX 

-X 

BOMBER 

4  GEN 

XX 

XX 

XX  XX 

XX 

.X 

3  GEN 

XX 

XX 

XX  XX 

XX 

.X 

BOMBER 

XX 

XX 

XX  XX 

XX 

.X 

4  GEN 

5  GEN 

XX 

XX 

XX  XX 

XX 

.X 

4  GEN 

XX 

XX 

XX  XX 

XX 

.X 

3  GEN 

XX 

XX 

XX  XX 

XX 

.X 

BOMBER 

XX 

XX 

XX  XX 

XX 

.X 

* 

* 

* 


a2a_escape_desire_classes 

3 

FIGHTER 

F15C  F15A  F14C  SU27  MiG31  MiG29  MiG23 


MULTI 

F15E  F16B50  end_typeaa 

BOMBER 

B1  SU24  end_typeaa 


MiG21  end_typeaa 
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escapeDesireDat (FRthreshold) by_phase, a2a_escape_desire_class, a2a_tactic 


FIGHTER 

AVERSE 

MEDIUM 

TOLERANT 

MULTI 

AVERSE 

MEDIUM 

TOLERANT 

BOMBER 

AVERSE 

MEDIUM 

TOLERANT 


FIGHTER 


MULTI 


BOMBER 


AVERSE 

XX 

MEDIUM 

XX 

TOLERANT 

XX 

AVERSE 

XX 

MEDIUM 

XX 

TOLERANT 

XX 

AVERSE 

XX 

MEDIUM 

XX 

TOLERANT 

XX 

FIGHTER 


MULTI 


BOMBER 


AVERSE 

XX 

MEDIUM 

XX 

TOLERANT 

XX 

AVERSE 

XX 

MEDIUM 

XX 

TOLERANT 

> 

XX 

AVERSE 

XX 

MEDIUM 

XX 

TOLERANT 

XX 
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